ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

26 Sep 2019Zhenzhong LanMingda ChenSebastian GoodmanKevin GimpelPiyush SharmaRadu Soricut

Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks. However, at some point further model increases become harder due to GPU/TPU memory limitations, longer training times, and unexpected model degradation... (read more)

PDF Abstract

Evaluation results from the paper


Task Dataset Model Metric name Metric value Global rank Compare
Linguistic Acceptability CoLA ALBERT Accuracy 69.1% # 1
Semantic Textual Similarity MRPC ALBERT Accuracy 93.4% # 1
Natural Language Inference MultiNLI ALBERT Matched 91.3 # 1
Natural Language Inference QNLI ALBERT Accuracy 99.2% # 1
Question Answering Quora Question Pairs ALBERT Accuracy 90.5% # 1
Natural Language Inference RTE ALBERT Accuracy 89.2% # 1
Question Answering SQuAD2.0 ALBERT (ensemble model) EM 89.731 # 1
Question Answering SQuAD2.0 ALBERT (ensemble model) F1 92.215 # 1
Question Answering SQuAD2.0 ALBERT (single model) EM 88.107 # 5
Question Answering SQuAD2.0 ALBERT (single model) F1 90.902 # 2
Question Answering SQuAD2.0 dev ALBERT xxlarge F1 88.1 # 3
Question Answering SQuAD2.0 dev ALBERT xxlarge EM 85.1 # 3
Question Answering SQuAD2.0 dev ALBERT xlarge F1 85.9 # 6
Question Answering SQuAD2.0 dev ALBERT xlarge EM 83.1 # 4
Question Answering SQuAD2.0 dev ALBERT large F1 82.1 # 8
Question Answering SQuAD2.0 dev ALBERT large EM 79 # 6
Question Answering SQuAD2.0 dev ALBERT base F1 79.1 # 10
Question Answering SQuAD2.0 dev ALBERT base EM 76.1 # 8
Sentiment Analysis SST-2 Binary classification ALBERT Accuracy 97.1 # 1
Semantic Textual Similarity STS Benchmark ALBERT Pearson Correlation 0.925 # 1
Natural Language Inference WNLI ALBERT Accuracy 91.8% # 1