RoBERTa: A Robustly Optimized BERT Pretraining Approach

Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Natural Language Inference ANLI test RoBERTa (Large) A1 72.4 # 2
A2 49.8 # 4
A3 44.4 # 4
Linguistic Acceptability CoLA RoBERTa Accuracy 67.8% # 10
Type prediction ManyTypes4TypeScript RoBERTa Average Accuracy 59.84 # 5
Average Precision 57.45 # 5
Average Recall 57.62 # 5
Average F1 57.54 # 5
Multi-task Language Understanding MMLU RoBERTa (fine-tuned) Humanities 27.9 # 11
Average (%) 27.9 # 11
Parameters (Billions) 0.354 # 2
STEM 27.0 # 12
Social Sciences 28.8 # 13
Other 27.7 # 13
Semantic Textual Similarity MRPC RoBERTa Accuracy 92.3% # 3
Natural Language Inference MultiNLI RoBERTa Matched 90.8 # 7
Mismatched 90.2 # 6
Natural Language Inference QNLI RoBERTa Accuracy 98.9% # 3
Question Answering Quora Question Pairs RoBERTa Accuracy 90.2% # 6
Reading Comprehension RACE RoBERTa Accuracy 83.2 # 7
Accuracy (High) 81.3 # 6
Accuracy (Middle) 86.5 # 6
Natural Language Inference RTE RoBERTa Accuracy 88.2% # 9
Document Image Classification RVL-CDIP Roberta base Accuracy 90.06 # 23
Parameters 125M # 16
Question Answering SQuAD2.0 RoBERTa (single model) EM 86.820 # 87
F1 89.795 # 87
Question Answering SQuAD2.0 dev RoBERTa (no data aug) F1 89.4 # 3
EM 86.5 # 3
Sentiment Analysis SST-2 Binary classification RoBERTa Accuracy 96.7 # 11
Semantic Textual Similarity STS Benchmark RoBERTa Pearson Correlation 0.922 # 7
Common Sense Reasoning SWAG RoBERTa Test 89.9 # 2
Natural Language Inference WNLI RoBERTa Accuracy 89% # 8

Results from Other Papers


Task Dataset Model Metric Name Metric Value Rank Uses Extra
Training Data
Source Paper Compare
Common Sense Reasoning CommonsenseQA RoBERTa Liu et al. (2019) Accuracy 72.1 # 11

Methods