TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Linguistic Acceptability	CoLA	ALBERT	Accuracy	69.1%	# 2
Semantic Textual Similarity	MRPC	ALBERT	Accuracy	93.4%	# 2
Natural Language Inference	MultiNLI	ALBERT	Matched	91.3	# 3
Natural Language Inference	QNLI	ALBERT	Accuracy	99.2%	# 1
Question Answering	Quora Question Pairs	ALBERT	Accuracy	90.5%	# 2
Natural Language Inference	RTE	ALBERT	Accuracy	89.2%	# 4
Question Answering	SQuAD2.0	ALBERT (ensemble model)	EM	89.731	# 19
Question Answering	SQuAD2.0	ALBERT (ensemble model)	F1	92.215	# 21
Question Answering	SQuAD2.0	ALBERT (single model)	EM	88.107	# 49
Question Answering	SQuAD2.0	ALBERT (single model)	F1	90.902	# 54
Question Answering	SQuAD2.0 dev	ALBERT xlarge	F1	85.9	# 7
Question Answering	SQuAD2.0 dev	ALBERT xlarge	EM	83.1	# 5
Question Answering	SQuAD2.0 dev	ALBERT xxlarge	F1	88.1	# 4
Question Answering	SQuAD2.0 dev	ALBERT xxlarge	EM	85.1	# 4
Question Answering	SQuAD2.0 dev	ALBERT large	F1	82.1	# 9
Question Answering	SQuAD2.0 dev	ALBERT large	EM	79	# 7
Question Answering	SQuAD2.0 dev	ALBERT base	F1	79.1	# 11
Question Answering	SQuAD2.0 dev	ALBERT base	EM	76.1	# 9
Sentiment Analysis	SST-2 Binary classification	ALBERT	Accuracy	97.1	# 3
Semantic Textual Similarity	STS Benchmark	ALBERT	Pearson Correlation	0.925	# 2
Natural Language Inference	WNLI	ALBERT	Accuracy	91.8%	# 5

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/albert-a-lite-bert-for-self-supervised/natural-language-inference-on-qnli)](https://paperswithcode.com/sota/natural-language-inference-on-qnli?p=albert-a-lite-bert-for-self-supervised)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/albert-a-lite-bert-for-self-supervised/linguistic-acceptability-on-cola)](https://paperswithcode.com/sota/linguistic-acceptability-on-cola?p=albert-a-lite-bert-for-self-supervised)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/albert-a-lite-bert-for-self-supervised/semantic-textual-similarity-on-mrpc)](https://paperswithcode.com/sota/semantic-textual-similarity-on-mrpc?p=albert-a-lite-bert-for-self-supervised)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/albert-a-lite-bert-for-self-supervised/question-answering-on-quora-question-pairs)](https://paperswithcode.com/sota/question-answering-on-quora-question-pairs?p=albert-a-lite-bert-for-self-supervised)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/albert-a-lite-bert-for-self-supervised/semantic-textual-similarity-on-sts-benchmark)](https://paperswithcode.com/sota/semantic-textual-similarity-on-sts-benchmark?p=albert-a-lite-bert-for-self-supervised)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/albert-a-lite-bert-for-self-supervised/natural-language-inference-on-multinli)](https://paperswithcode.com/sota/natural-language-inference-on-multinli?p=albert-a-lite-bert-for-self-supervised)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/albert-a-lite-bert-for-self-supervised/sentiment-analysis-on-sst-2-binary)](https://paperswithcode.com/sota/sentiment-analysis-on-sst-2-binary?p=albert-a-lite-bert-for-self-supervised)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/albert-a-lite-bert-for-self-supervised/natural-language-inference-on-rte)](https://paperswithcode.com/sota/natural-language-inference-on-rte?p=albert-a-lite-bert-for-self-supervised)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/albert-a-lite-bert-for-self-supervised/question-answering-on-squad20-dev)](https://paperswithcode.com/sota/question-answering-on-squad20-dev?p=albert-a-lite-bert-for-self-supervised)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/albert-a-lite-bert-for-self-supervised/natural-language-inference-on-wnli)](https://paperswithcode.com/sota/natural-language-inference-on-wnli?p=albert-a-lite-bert-for-self-supervised)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/albert-a-lite-bert-for-self-supervised/question-answering-on-squad20)](https://paperswithcode.com/sota/question-answering-on-squad20?p=albert-a-lite-bert-for-self-supervised)`

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

ICLR 2020 • Zhenzhong Lan • Mingda Chen • Sebastian Goodman • Kevin Gimpel • Piyush Sharma • Radu Soricut

Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks. However, at some point further model increases become harder due to GPU/TPU memory limitations and longer training times... (read more)

PDF Abstract ICLR 2020 PDF ICLR 2020 Abstract

Code

Add Remove Mark official

google-research/ALBERT official

2,645

tensorflow/models

69,487

huggingface/transformers

43,811

brightmart/albert_zh

3,230

CLUEbenchmark/CLUE

1,799

See all 35 implementations

Tasks

Add Remove

LINGUISTIC ACCEPTABILITY

NATURAL LANGUAGE INFERENCE

QUESTION ANSWERING

SELF-SUPERVISED LEARNING

SEMANTIC TEXTUAL SIMILARITY

Datasets

SQuAD

GLUE

SST

MultiNLI

QNLI

RACE

MRPC

SuperGLUE

CoLA

Quora

Results from the Paper

Edit

Ranked #1 on Natural Language Inference on QNLI

Get a GitHub badge

TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK	BENCHMARK
Linguistic Acceptability	CoLA	ALBERT	Accuracy	69.1%	# 2	Compare
Semantic Textual Similarity	MRPC	ALBERT	Accuracy	93.4%	# 2	Compare
Natural Language Inference	MultiNLI	ALBERT	Matched	91.3	# 3	Compare
Natural Language Inference	QNLI	ALBERT	Accuracy	99.2%	# 1	Compare
Question Answering	Quora Question Pairs	ALBERT	Accuracy	90.5%	# 2	Compare
Natural Language Inference	RTE	ALBERT	Accuracy	89.2%	# 4	Compare
Question Answering	SQuAD2.0	ALBERT (ensemble model)	EM	89.731	# 19	Compare
Question Answering	SQuAD2.0	ALBERT (ensemble model)	F1	92.215	# 21	Compare
Question Answering	SQuAD2.0	ALBERT (single model)	EM	88.107	# 49	Compare
Question Answering	SQuAD2.0	ALBERT (single model)	F1	90.902	# 54	Compare
Question Answering	SQuAD2.0 dev	ALBERT xlarge	F1	85.9	# 7	Compare
Question Answering	SQuAD2.0 dev	ALBERT xlarge	EM	83.1	# 5	Compare
Question Answering	SQuAD2.0 dev	ALBERT xxlarge	F1	88.1	# 4	Compare
Question Answering	SQuAD2.0 dev	ALBERT xxlarge	EM	85.1	# 4	Compare
Question Answering	SQuAD2.0 dev	ALBERT large	F1	82.1	# 9	Compare
Question Answering	SQuAD2.0 dev	ALBERT large	EM	79	# 7	Compare
Question Answering	SQuAD2.0 dev	ALBERT base	F1	79.1	# 11	Compare
Question Answering	SQuAD2.0 dev	ALBERT base	EM	76.1	# 9	Compare
Sentiment Analysis	SST-2 Binary classification	ALBERT	Accuracy	97.1	# 3	Compare
Semantic Textual Similarity	STS Benchmark	ALBERT	Pearson Correlation	0.925	# 2	Compare
Natural Language Inference	WNLI	ALBERT	Accuracy	91.8%	# 5	Compare

Methods used in the Paper

Add Remove

METHOD	TYPE
Weight Decay	Regularization
Dropout	Regularization
Attention Dropout	Regularization
Linear Warmup With Linear Decay	Learning Rate Schedules
BERT	Language Models
Residual Connection	Skip Connections
Adam	Stochastic Optimization
LAMB	Large Batch Optimization
GELU	Activation Functions
Dense Connections	Feedforward Networks
Multi-Head Attention	Attention Modules
WordPiece	Subword Segmentation
Softmax	Output Functions
Layer Normalization	Normalization
Scaled Dot-Product Attention	Attention Mechanisms
ALBERT	Transformers

Edit Social Preview

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations Edit social preview

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods used in the Paper Edit Add Remove