RoBERTa: A Robustly Optimized BERT Pretraining Approach

Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results... We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code. read more

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Natural Language Inference ANLI test RoBERTa (Large) A1 72.4 # 2
A2 49.8 # 4
A3 44.4 # 4
Linguistic Acceptability CoLA RoBERTa Accuracy 67.8% # 9
Semantic Textual Similarity MRPC RoBERTa Accuracy 92.3% # 3
Natural Language Inference MultiNLI RoBERTa Matched 90.8 # 7
Mismatched 90.2 # 6
Natural Language Inference QNLI RoBERTa Accuracy 98.9% # 3
Question Answering Quora Question Pairs RoBERTa Accuracy 90.2% # 6
Reading Comprehension RACE RoBERTa Accuracy 83.2 # 8
Accuracy (High) 81.3 # 6
Accuracy (Middle) 86.5 # 6
Natural Language Inference RTE RoBERTa Accuracy 88.2% # 7
Question Answering SQuAD2.0 RoBERTa (single model) EM 86.820 # 82
F1 89.795 # 81
Question Answering SQuAD2.0 dev RoBERTa (no data aug) F1 89.4 # 3
EM 86.5 # 3
Sentiment Analysis SST-2 Binary classification RoBERTa Accuracy 96.7 # 10
Semantic Textual Similarity STS Benchmark RoBERTa Pearson Correlation 0.922 # 6
Common Sense Reasoning SWAG RoBERTa Test 89.9 # 2
Natural Language Inference WNLI RoBERTa Accuracy 89% # 8

Results from Other Papers


Task Dataset Model Metric Name Metric Value Rank Uses Extra
Training Data
Source Paper Compare
Common Sense Reasoning CommonsenseQA RoBERTa Liu et al. (2019) Accuracy 72.1 # 6

Methods