XLNet: Generalized Autoregressive Pretraining for Language Understanding

NeurIPS 2019  ·  Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le ·

With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into pretraining. Empirically, under comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by a large margin, including question answering, natural language inference, sentiment analysis, and document ranking.

PDF Abstract NeurIPS 2019 PDF NeurIPS 2019 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data 		Result Benchmark
Humor Detection 200k Short Texts for Humor Detection XLNet Large Cased F1-score 0.920 # 2
Compare
Text Classification AG News XLNet Error 4.45 # 1
Compare
Text Classification Amazon-2 XLNet Error 2.11 # 1
Compare
Text Classification Amazon-5 XLNet Error 31.67 # 1
Compare
Natural Language Inference ANLI test XLNet (Large) A1 70.3 # 7
Compare
A2 50.9 # 12
Compare
A3 49.4 # 13
Compare
Linguistic Acceptability CoLA XLNet (single model) Accuracy 69% # 11
Compare
Text Classification DBpedia XLNet Error 0.62 # 1
Compare
Sentiment Analysis IMDb XLNet Accuracy 96.21 # 1
Compare
Text Classification IMDb XLNet Accuracy (2 classes) 96.8 # 2
Compare
Accuracy (10 classes) - # 3
Compare
Semantic Textual Similarity MRPC XLNet (single model) Accuracy 90.8% # 9
Compare
Natural Language Inference MultiNLI XLNet (single model) Matched 90.8 # 7
Compare
Natural Language Inference QNLI XLNet (single model) Accuracy 94.9% # 11
Compare
Question Answering Quora Question Pairs XLNet (single model) Accuracy 92.3% # 1
Compare
Reading Comprehension RACE XLNet Accuracy (High) 84.0 # 5
Compare
Accuracy (Middle) 88.6 # 5
Compare
Audio Question Answering RoadTracer XLNet RACE-h 80.21 # 1
Compare
Chinese Reading Comprehension RoadTracer XLNet Accuracy 85.4 # 1
Compare
Natural Language Inference RTE XLNet (single model) Accuracy 85.9% # 14
Compare
Semantic Textual Similarity SentEval XLNet-Large MRPC 93.0/90.7 # 1
Compare
SICK-R - # 3
Compare
SICK-E - # 3
Compare
STS 91.6/91.1* # 1
Compare
Question Answering SQuAD1.1 XLNet (single model) EM 89.898 # 5
Compare
F1 95.080 # 5
Compare
Hardware Burden 46449G # 1
Compare
Question Answering SQuAD1.1 dev XLNet (single model) EM 89.7 # 4
Compare
F1 95.1 # 3
Compare
Question Answering SQuAD2.0 XLNet (single model) EM 87.926 # 77
Compare
F1 90.689 # 79
Compare
Question Answering SQuAD2.0 dev XLNet (single model) F1 90.6 # 1
Compare
EM 87.9 # 1
Compare
Sentiment Analysis SST-2 Binary classification XLNet (single model) Accuracy 97 # 7
Compare
Sentiment Analysis SST-2 Binary classification XLNet-Large (ensemble) Accuracy 96.8 # 10
Compare
Semantic Textual Similarity STS Benchmark XLNet (single model) Pearson Correlation 0.925 # 4
Compare
Natural Language Inference WNLI XLNet Accuracy 92.5% # 4
Compare
Text Classification Yelp-2 XLNet Accuracy 98.63% # 1
Compare
Text Classification Yelp-5 XLNet Accuracy 72.95% # 2
Compare
Sentiment Analysis Yelp Binary classification XLNet Error 1.37 # 1
Compare
Sentiment Analysis Yelp Fine-grained classification XLNet Error 27.05 # 1
Compare

Methods
Add Remove

Contact us on: hello@paperswithcode.com . Papers With Code is a free resource with all data licensed under CC-BY-SA.
Terms Data policy Cookies policy