RoBERTa: A Robustly Optimized BERT Pretraining Approach

26 Jul 2019Yinhan LiuMyle OttNaman GoyalJingfei DuMandar JoshiDanqi ChenOmer LevyMike LewisLuke ZettlemoyerVeselin Stoyanov

Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results... (read more)

PDF Abstract

Evaluation results from the paper

 SOTA for Question Answering on SQuAD2.0 dev (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric name Metric value Global rank Uses extra
training data
Linguistic Acceptability CoLA RoBERTa Accuracy 67.8% # 4
Semantic Textual Similarity MRPC RoBERTa Accuracy 92.3% # 3
Natural Language Inference MultiNLI RoBERTa Matched 90.8 # 2
Natural Language Inference MultiNLI RoBERTa Mismatched 90.2 # 1
Natural Language Inference QNLI RoBERTa Accuracy 98.9% # 2
Question Answering Quora Question Pairs RoBERTa Accuracy 90.2% # 3
Reading Comprehension RACE RoBERTa Accuracy 83.2 # 1
Natural Language Inference RTE RoBERTa Accuracy 88.2% # 2
Question Answering SQuAD2.0 RoBERTa EM 86.820 # 11
Question Answering SQuAD2.0 RoBERTa F1 89.795 # 10
Question Answering SQuAD2.0 dev RoBERTa (no data aug) F1 89.4 # 1
Question Answering SQuAD2.0 dev RoBERTa (no data aug) EM 86.5 # 1
Sentiment Analysis SST-2 Binary classification RoBERTa Accuracy 96.7 # 4
Semantic Textual Similarity STS Benchmark RoBERTa Pearson Correlation 0.922 # 2
Natural Language Inference WNLI RoBERTa Accuracy 89.0% # 3