ERNIE 2.0: A Continual Pre-training Framework for Language Understanding

29 Jul 2019Yu SunShuohuan WangYukun LiShikun FengHao TianHua WuHaifeng Wang

Recently, pre-trained models have achieved state-of-the-art results in various language understanding tasks, which indicates that pre-training on large-scale corpora may play a crucial role in natural language processing. Current pre-training procedures usually focus on training the model with several simple tasks to grasp the co-occurrence of words or sentences... (read more)

PDF Abstract

Evaluation results from the paper


Task Dataset Model Metric name Metric value Global rank Compare
Linguistic Acceptability CoLA ERNIE 2.0 Base Accuracy 55.2% # 4
Linguistic Acceptability CoLA ERNIE 2.0 Large Accuracy 63.5% # 3
Semantic Textual Similarity MRPC ERNIE 2.0 Base Accuracy 86.1% # 4
Semantic Textual Similarity MRPC ERNIE 2.0 Large Accuracy 87.4% # 3
Natural Language Inference MultiNLI ERNIE 2.0 Base Matched 86.1 # 5
Natural Language Inference MultiNLI ERNIE 2.0 Base Mismatched 85.5 # 5
Natural Language Inference MultiNLI ERNIE 2.0 Large Matched 88.7 # 3
Natural Language Inference MultiNLI ERNIE 2.0 Large Mismatched 88.8 # 3
Natural Language Inference QNLI ERNIE 2.0 Large Accuracy 94.6% # 3
Natural Language Inference QNLI ERNIE 2.0 Base Accuracy 92.9% # 4
Question Answering Quora Question Pairs ERNIE 2.0 Large Accuracy 90.1% # 3
Question Answering Quora Question Pairs ERNIE 2.0 Base Accuracy 89.8% # 4
Natural Language Inference RTE ERNIE 2.0 Base Accuracy 74.8% # 4
Natural Language Inference RTE ERNIE 2.0 Large Accuracy 80.2% # 3
Sentiment Analysis SST-2 Binary classification ERNIE 2.0 Large Accuracy 96.0 # 3
Sentiment Analysis SST-2 Binary classification ERNIE 2.0 Base Accuracy 95.0 # 5
Semantic Textual Similarity STS Benchmark ERNIE 2.0 Large Pearson Correlation 0,912 # 4
Semantic Textual Similarity STS Benchmark ERNIE 2.0 Base Pearson Correlation 0.876 # 2
Natural Language Inference WNLI ERNIE 2.0 Base Accuracy 65.1% # 4
Natural Language Inference WNLI ERNIE 2.0 Large Accuracy 67.8% # 3