TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Question Answering	BoolQ	T5-Small 60M (fine-tuned)	Accuracy	76.4	# 31
Question Answering	BoolQ	T5-Base 220M (fine-tuned)	Accuracy	81.4	# 25
Question Answering	BoolQ	T5-XXL 11B (fine-tuned)	Accuracy	91.2	# 4
Question Answering	BoolQ	T5-Large 770M (fine-tuned)	Accuracy	85.4	# 15
Document Summarization	CNN / Daily Mail	T5-11B	ROUGE-1	43.52	# 11
Document Summarization	CNN / Daily Mail	T5-11B	ROUGE-2	21.55	# 2
Document Summarization	CNN / Daily Mail	T5-11B	ROUGE-L	40.69	# 8
Abstractive Text Summarization	CNN / Daily Mail	T5	ROUGE-1	43.52	# 22
Abstractive Text Summarization	CNN / Daily Mail	T5	ROUGE-2	21.55	# 7
Abstractive Text Summarization	CNN / Daily Mail	T5	ROUGE-L	40.69	# 22
Linguistic Acceptability	CoLA	T5-Large 770M	Accuracy	61.2%	# 28
Linguistic Acceptability	CoLA	T5-11B	Accuracy	70.8%	# 12
Linguistic Acceptability	CoLA	T5-XL 3B	Accuracy	67.1%	# 22
Linguistic Acceptability	CoLA	T5-Base	Accuracy	51.1%	# 37
Linguistic Acceptability	CoLA	T5-Small	Accuracy	41.0%	# 41
Natural Language Inference	CommitmentBank	T5-Base 220M (fine-tuned)	F1	86.2	# 7
Natural Language Inference	CommitmentBank	T5-Base 220M (fine-tuned)	Accuracy	94	# 9
Natural Language Inference	CommitmentBank	T5-XXL 11B (fine-tuned)	F1	93.9	# 5
Natural Language Inference	CommitmentBank	T5-XXL 11B (fine-tuned)	Accuracy	96.8	# 7
Natural Language Inference	CommitmentBank	T5-Large 770M (fine-tuned)	F1	90.3	# 6
Natural Language Inference	CommitmentBank	T5-Large 770M (fine-tuned)	Accuracy	94.4	# 8
Question Answering	COPA	T5-XXL 11B (fine-tuned)	Accuracy	94.8	# 9
Question Answering	COPA	T5-Base 220M (fine-tuned)	Accuracy	71.2	# 47
Question Answering	COPA	T5-Large 770M (fine-tuned)	Accuracy	83.4	# 33
Question Answering	COPA	T5-XL 3B (fine-tuned)	Accuracy	92	# 11
Semantic Textual Similarity	MRPC	T5-Small	Accuracy	86.6%	# 31
Semantic Textual Similarity	MRPC	T5-Small	F1	89.7	# 11
Semantic Textual Similarity	MRPC	T5-11B	Accuracy	90.0%	# 15
Semantic Textual Similarity	MRPC	T5-11B	F1	91.9	# 4
Semantic Textual Similarity	MRPC	T5-Large	Accuracy	89.9%	# 16
Semantic Textual Similarity	MRPC	T5-Large	F1	92.4	# 3
Semantic Textual Similarity	MRPC	T5-Base	Accuracy	87.5%	# 25
Semantic Textual Similarity	MRPC	T5-Base	F1	90.7	# 10
Semantic Textual Similarity	MRPC	T5-3B	Accuracy	89.2%	# 19
Semantic Textual Similarity	MRPC	T5-3B	F1	92.5	# 2
Natural Language Inference	MultiNLI	T5-Large	Matched	89.9	# 11
Natural Language Inference	MultiNLI	T5-XXL 11B (fine-tuned)	Matched	92.0	# 2
Natural Language Inference	MultiNLI	T5-3B	Matched	91.4	# 4
Natural Language Inference	MultiNLI	T5-3B	Mismatched	91.2	# 4
Natural Language Inference	MultiNLI	T5-Base	Matched	87.1	# 21
Natural Language Inference	MultiNLI	T5-Base	Mismatched	86.2	# 15
Natural Language Inference	MultiNLI	T5-Small	Matched	82.4	# 36
Natural Language Inference	MultiNLI	T5-Small	Mismatched	82.3	# 25
Natural Language Inference	MultiNLI	T5-Large 770M	Mismatched	89.6	# 8
Natural Language Inference	MultiNLI	T5-11B	Mismatched	91.7	# 2
Question Answering	MultiRC	T5-XXL 11B (fine-tuned)	F1	88.1	# 7
Question Answering	MultiRC	T5-11B	EM	63.3	# 3
Multimodal Intent Recognition	PhotoChat	T5-base	F1	58.1	# 3
Multimodal Intent Recognition	PhotoChat	T5-base	Precision	58.2	# 2
Multimodal Intent Recognition	PhotoChat	T5-base	Recall	57.9	# 5
Multimodal Intent Recognition	PhotoChat	T5-3B	F1	58.9	# 2
Multimodal Intent Recognition	PhotoChat	T5-3B	Precision	54.1	# 5
Multimodal Intent Recognition	PhotoChat	T5-3B	Recall	64.6	# 2
Natural Language Inference	QNLI	T5-Base	Accuracy	93.7%	# 19
Natural Language Inference	QNLI	T5-Small	Accuracy	90.3%	# 35
Natural Language Inference	QNLI	T5-3B	Accuracy	96.3%	# 7
Natural Language Inference	QNLI	T5-11B	Accuracy	96.7%	# 6
Natural Language Inference	QNLI	T5-Large 770M	Accuracy	94.8%	# 12
Question Answering	Quora Question Pairs	T5-Large 770M	Accuracy	89.9%	# 9
Question Answering	Quora Question Pairs	T5-11B	Accuracy	90.4%	# 4
Question Answering	Quora Question Pairs	T5-Small	Accuracy	88.0%	# 16
Question Answering	Quora Question Pairs	T5-Base	Accuracy	89.4%	# 12
Question Answering	Quora Question Pairs	T5-3B	Accuracy	89.7%	# 11
Common Sense Reasoning	ReCoRD	T5-11B	F1	94.1	# 5
Common Sense Reasoning	ReCoRD	T5-XXL 11B (fine-tuned)	EM	93.4	# 6
Natural Language Inference	RTE	T5-Small	Accuracy	69.9%	# 54
Natural Language Inference	RTE	T5-XXL 11B (fine-tuned)	Accuracy	92.5%	# 8
Natural Language Inference	RTE	T5-Base 220M	Accuracy	80.1%	# 36
Natural Language Inference	RTE	T5-XL 3B	Accuracy	91.1%	# 14
Natural Language Inference	RTE	T5-Large 770M	Accuracy	87.2%	# 21
Question Answering	SQuAD1.1 dev	T5-Large 770M	EM	86.66	# 6
Question Answering	SQuAD1.1 dev	T5-Large 770M	F1	93.79	# 6
Question Answering	SQuAD1.1 dev	T5-Base	EM	85.44	# 8
Question Answering	SQuAD1.1 dev	T5-Base	F1	92.08	# 8
Question Answering	SQuAD1.1 dev	T5-3B	EM	88.53	# 5
Question Answering	SQuAD1.1 dev	T5-3B	F1	94.95	# 5
Question Answering	SQuAD1.1 dev	T5-Small	EM	79.1	# 16
Question Answering	SQuAD1.1 dev	T5-Small	F1	87.24	# 18
Question Answering	SQuAD1.1 dev	T5-11B	EM	90.06	# 1
Question Answering	SQuAD1.1 dev	T5-11B	F1	95.64	# 2
Sentiment Analysis	SST-2 Binary classification	T5-11B	Accuracy	97.5	# 1
Sentiment Analysis	SST-2 Binary classification	T5-Large 770M	Accuracy	96.3	# 17
Sentiment Analysis	SST-2 Binary classification	T5-3B	Accuracy	97.4	# 3
Sentiment Analysis	SST-2 Binary classification	T5-Base	Accuracy	95.2	# 24
Sentiment Analysis	SST-2 Binary classification	T5-Small	Accuracy	91.8	# 47
Semantic Textual Similarity	STS Benchmark	T5-11B	Pearson Correlation	0.925	# 4
Semantic Textual Similarity	STS Benchmark	T5-11B	Spearman Correlation	0.921	# 4
Semantic Textual Similarity	STS Benchmark	T5-Large 770M	Spearman Correlation	0.886	# 12
Semantic Textual Similarity	STS Benchmark	T5-Small	Pearson Correlation	0.856	# 25
Semantic Textual Similarity	STS Benchmark	T5-Small	Spearman Correlation	0.85	# 24
Semantic Textual Similarity	STS Benchmark	T5-Base	Pearson Correlation	0.894	# 22
Semantic Textual Similarity	STS Benchmark	T5-Large	Pearson Correlation	0.899	# 20
Semantic Textual Similarity	STS Benchmark	T5-3B	Pearson Correlation	0.906	# 17
Semantic Textual Similarity	STS Benchmark	T5-3B	Spearman Correlation	0.898	# 6
Question Answering	WebQuestions	T5.1.1-XXL+SSM	EM	42.8	# 6
Semantic Parsing	WebQuestionsSP	T5-11B (Raffel et al., 2020)	Accuracy	56.5	# 5
Poll Generation	WeiboPolls	T5	ROUGE-1	45.33	# 2
Poll Generation	WeiboPolls	T5	ROUGE-L	42.69	# 2
Poll Generation	WeiboPolls	T5	BLEU-1	37.34	# 2
Poll Generation	WeiboPolls	T5	BLEU-3	21.06	# 2
Answer Generation	WeiboPolls	T5	ROUGE-1	46.20	# 2
Answer Generation	WeiboPolls	T5	ROUGE-L	43.32	# 2
Answer Generation	WeiboPolls	T5	BLEU-1	37.77	# 2
Answer Generation	WeiboPolls	T5	BLEU-3	25.86	# 1
Question Generation	WeiboPolls	T5	ROUGE-1	44.46	# 2
Question Generation	WeiboPolls	T5	ROUGE-L	42.06	# 2
Question Generation	WeiboPolls	T5	BLEU-1	36.91	# 2
Question Generation	WeiboPolls	T5	BLEU-3	16.26	# 2
Coreference Resolution	Winograd Schema Challenge	T5-XXL 11B (fine-tuned)	Accuracy	93.8	# 7
Machine Translation	WMT2014 English-French	T5	BLEU score	43.4	# 9
Machine Translation	WMT2014 English-German	T5-11B	BLEU score	32.1	# 4
Machine Translation	WMT2014 English-German	T5-11B	Number of Params	11110M	# 1
Natural Language Inference	WNLI	T5-Base 220M	Accuracy	78.8	# 12
Natural Language Inference	WNLI	T5-Small 60M	Accuracy	69.2	# 18
Natural Language Inference	WNLI	T5-XXL 11B	Accuracy	93.2	# 3
Natural Language Inference	WNLI	T5-XL 3B	Accuracy	89.7	# 6
Natural Language Inference	WNLI	T5-Large 770M	Accuracy	85.6	# 10
Word Sense Disambiguation	Words in Context	T5-XXL 11B	Accuracy	76.9	# 8

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/exploring-the-limits-of-transfer-learning/question-answering-on-squad11-dev)](https://paperswithcode.com/sota/question-answering-on-squad11-dev?p=exploring-the-limits-of-transfer-learning)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/exploring-the-limits-of-transfer-learning/sentiment-analysis-on-sst-2-binary)](https://paperswithcode.com/sota/sentiment-analysis-on-sst-2-binary?p=exploring-the-limits-of-transfer-learning)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/exploring-the-limits-of-transfer-learning/natural-language-inference-on-multinli)](https://paperswithcode.com/sota/natural-language-inference-on-multinli?p=exploring-the-limits-of-transfer-learning)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/exploring-the-limits-of-transfer-learning/multimodal-intent-recognition-on-photochat)](https://paperswithcode.com/sota/multimodal-intent-recognition-on-photochat?p=exploring-the-limits-of-transfer-learning)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/exploring-the-limits-of-transfer-learning/poll-generation-on-weibopolls)](https://paperswithcode.com/sota/poll-generation-on-weibopolls?p=exploring-the-limits-of-transfer-learning)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/exploring-the-limits-of-transfer-learning/answer-generation-on-weibopolls)](https://paperswithcode.com/sota/answer-generation-on-weibopolls?p=exploring-the-limits-of-transfer-learning)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/exploring-the-limits-of-transfer-learning/question-generation-on-weibopolls)](https://paperswithcode.com/sota/question-generation-on-weibopolls?p=exploring-the-limits-of-transfer-learning)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/exploring-the-limits-of-transfer-learning/question-answering-on-multirc)](https://paperswithcode.com/sota/question-answering-on-multirc?p=exploring-the-limits-of-transfer-learning)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/exploring-the-limits-of-transfer-learning/natural-language-inference-on-wnli)](https://paperswithcode.com/sota/natural-language-inference-on-wnli?p=exploring-the-limits-of-transfer-learning)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/exploring-the-limits-of-transfer-learning/question-answering-on-boolq)](https://paperswithcode.com/sota/question-answering-on-boolq?p=exploring-the-limits-of-transfer-learning)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/exploring-the-limits-of-transfer-learning/question-answering-on-quora-question-pairs)](https://paperswithcode.com/sota/question-answering-on-quora-question-pairs?p=exploring-the-limits-of-transfer-learning)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/exploring-the-limits-of-transfer-learning/semantic-textual-similarity-on-sts-benchmark)](https://paperswithcode.com/sota/semantic-textual-similarity-on-sts-benchmark?p=exploring-the-limits-of-transfer-learning)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/exploring-the-limits-of-transfer-learning/machine-translation-on-wmt2014-english-german)](https://paperswithcode.com/sota/machine-translation-on-wmt2014-english-german?p=exploring-the-limits-of-transfer-learning)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/exploring-the-limits-of-transfer-learning/common-sense-reasoning-on-record)](https://paperswithcode.com/sota/common-sense-reasoning-on-record?p=exploring-the-limits-of-transfer-learning)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/exploring-the-limits-of-transfer-learning/semantic-parsing-on-webquestionssp)](https://paperswithcode.com/sota/semantic-parsing-on-webquestionssp?p=exploring-the-limits-of-transfer-learning)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/exploring-the-limits-of-transfer-learning/natural-language-inference-on-qnli)](https://paperswithcode.com/sota/natural-language-inference-on-qnli?p=exploring-the-limits-of-transfer-learning)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/exploring-the-limits-of-transfer-learning/question-answering-on-webquestions)](https://paperswithcode.com/sota/question-answering-on-webquestions?p=exploring-the-limits-of-transfer-learning)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/exploring-the-limits-of-transfer-learning/natural-language-inference-on-commitmentbank)](https://paperswithcode.com/sota/natural-language-inference-on-commitmentbank?p=exploring-the-limits-of-transfer-learning)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/exploring-the-limits-of-transfer-learning/coreference-resolution-on-winograd-schema)](https://paperswithcode.com/sota/coreference-resolution-on-winograd-schema?p=exploring-the-limits-of-transfer-learning)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/exploring-the-limits-of-transfer-learning/natural-language-inference-on-rte)](https://paperswithcode.com/sota/natural-language-inference-on-rte?p=exploring-the-limits-of-transfer-learning)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/exploring-the-limits-of-transfer-learning/word-sense-disambiguation-on-words-in-context)](https://paperswithcode.com/sota/word-sense-disambiguation-on-words-in-context?p=exploring-the-limits-of-transfer-learning)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/exploring-the-limits-of-transfer-learning/question-answering-on-copa)](https://paperswithcode.com/sota/question-answering-on-copa?p=exploring-the-limits-of-transfer-learning)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/exploring-the-limits-of-transfer-learning/machine-translation-on-wmt2014-english-french)](https://paperswithcode.com/sota/machine-translation-on-wmt2014-english-french?p=exploring-the-limits-of-transfer-learning)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/exploring-the-limits-of-transfer-learning/document-summarization-on-cnn-daily-mail)](https://paperswithcode.com/sota/document-summarization-on-cnn-daily-mail?p=exploring-the-limits-of-transfer-learning)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/exploring-the-limits-of-transfer-learning/linguistic-acceptability-on-cola)](https://paperswithcode.com/sota/linguistic-acceptability-on-cola?p=exploring-the-limits-of-transfer-learning)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/exploring-the-limits-of-transfer-learning/semantic-textual-similarity-on-mrpc)](https://paperswithcode.com/sota/semantic-textual-similarity-on-mrpc?p=exploring-the-limits-of-transfer-learning)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/exploring-the-limits-of-transfer-learning/abstractive-text-summarization-on-cnn-daily)](https://paperswithcode.com/sota/abstractive-text-summarization-on-cnn-daily?p=exploring-the-limits-of-transfer-learning)`

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

arXiv 2019 · Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu ·

Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.

PDF Abstract

Code

Add Remove Mark official

huggingface/transformers

124,593

PaddlePaddle/PaddleNLP

11,384

google-research/text-to-text-transf…

↳ Quickstart in

Colab

5,893

google-research/t5x

2,479

amazon-science/chronos-forecasting

1,566

See all 51 implementations

Tasks

Add Remove

Answer Generation

Common Sense Reasoning

Coreference Resolution

Linguistic Acceptability

Machine Translation

Multimodal Intent Recognition

Natural Language Inference

Poll Generation

Question Answering

Question Generation

Semantic Parsing

Semantic Textual Similarity

Sentiment Analysis

Transfer Learning

Word Sense Disambiguation

Datasets

Introduced in the Paper:

Used in the Paper:

GLUE

SST

SQuAD

MultiNLI SST-2

QNLI

MRPC

CoLA

CNN/Daily Mail

BoolQ

WebText

WSC

COPA

WMT 2014

WebQuestions

MultiRC

ReCoRD

Quora WebQuestionsSP

Quora Question Pairs RTE STS Benchmark

PhotoChat WNLI CommitmentBank

WeiboPolls

Results from the Paper

Edit

Ranked #1 on Sentiment Analysis on SST-2 Binary classification

Get a GitHub badge

Task	Dataset	Model	Metric Name	Metric Value	Global Rank	Benchmark
Question Answering	BoolQ	T5-Small 60M (fine-tuned)	Accuracy	76.4	# 31	Compare
Question Answering	BoolQ	T5-Base 220M (fine-tuned)	Accuracy	81.4	# 25	Compare
Question Answering	BoolQ	T5-XXL 11B (fine-tuned)	Accuracy	91.2	# 4	Compare
Question Answering	BoolQ	T5-Large 770M (fine-tuned)	Accuracy	85.4	# 15	Compare
Document Summarization	CNN / Daily Mail	T5-11B	ROUGE-1	43.52	# 11	Compare
			ROUGE-2	21.55	# 2	Compare
			ROUGE-L	40.69	# 8	Compare
Abstractive Text Summarization	CNN / Daily Mail	T5	ROUGE-1	43.52	# 22	Compare
			ROUGE-2	21.55	# 7	Compare
			ROUGE-L	40.69	# 22	Compare
Linguistic Acceptability	CoLA	T5-Large 770M	Accuracy	61.2%	# 28	Compare
Linguistic Acceptability	CoLA	T5-11B	Accuracy	70.8%	# 12	Compare
Linguistic Acceptability	CoLA	T5-XL 3B	Accuracy	67.1%	# 22	Compare
Linguistic Acceptability	CoLA	T5-Base	Accuracy	51.1%	# 37	Compare
Linguistic Acceptability	CoLA	T5-Small	Accuracy	41.0%	# 41	Compare
Natural Language Inference	CommitmentBank	T5-Base 220M (fine-tuned)	F1	86.2	# 7	Compare
Natural Language Inference	CommitmentBank	T5-Base 220M (fine-tuned)	Accuracy	94	# 9	Compare
Natural Language Inference	CommitmentBank	T5-XXL 11B (fine-tuned)	F1	93.9	# 5	Compare
Natural Language Inference	CommitmentBank	T5-XXL 11B (fine-tuned)	Accuracy	96.8	# 7	Compare
Natural Language Inference	CommitmentBank	T5-Large 770M (fine-tuned)	F1	90.3	# 6	Compare
Natural Language Inference	CommitmentBank	T5-Large 770M (fine-tuned)	Accuracy	94.4	# 8	Compare
Question Answering	COPA	T5-XXL 11B (fine-tuned)	Accuracy	94.8	# 9	Compare
Question Answering	COPA	T5-Base 220M (fine-tuned)	Accuracy	71.2	# 47	Compare
Question Answering	COPA	T5-Large 770M (fine-tuned)	Accuracy	83.4	# 33	Compare
Question Answering	COPA	T5-XL 3B (fine-tuned)	Accuracy	92	# 11	Compare
Semantic Textual Similarity	MRPC	T5-Small	Accuracy	86.6%	# 31	Compare
Semantic Textual Similarity	MRPC	T5-Small	F1	89.7	# 11	Compare
Semantic Textual Similarity	MRPC	T5-11B	Accuracy	90.0%	# 15	Compare
Semantic Textual Similarity	MRPC	T5-11B	F1	91.9	# 4	Compare
Semantic Textual Similarity	MRPC	T5-Large	Accuracy	89.9%	# 16	Compare
Semantic Textual Similarity	MRPC	T5-Large	F1	92.4	# 3	Compare
Semantic Textual Similarity	MRPC	T5-Base	Accuracy	87.5%	# 25	Compare
Semantic Textual Similarity	MRPC	T5-Base	F1	90.7	# 10	Compare
Semantic Textual Similarity	MRPC	T5-3B	Accuracy	89.2%	# 19	Compare
Semantic Textual Similarity	MRPC	T5-3B	F1	92.5	# 2	Compare
Natural Language Inference	MultiNLI	T5-Large	Matched	89.9	# 11	Compare
Natural Language Inference	MultiNLI	T5-XXL 11B (fine-tuned)	Matched	92.0	# 2	Compare
Natural Language Inference	MultiNLI	T5-3B	Matched	91.4	# 4	Compare
Natural Language Inference	MultiNLI	T5-3B	Mismatched	91.2	# 4	Compare
Natural Language Inference	MultiNLI	T5-Base	Matched	87.1	# 21	Compare
Natural Language Inference	MultiNLI	T5-Base	Mismatched	86.2	# 15	Compare
Natural Language Inference	MultiNLI	T5-Small	Matched	82.4	# 36	Compare
Natural Language Inference	MultiNLI	T5-Small	Mismatched	82.3	# 25	Compare
Natural Language Inference	MultiNLI	T5-Large 770M	Mismatched	89.6	# 8	Compare
Natural Language Inference	MultiNLI	T5-11B	Mismatched	91.7	# 2	Compare
Question Answering	MultiRC	T5-XXL 11B (fine-tuned)	F1	88.1	# 7	Compare
Question Answering	MultiRC	T5-11B	EM	63.3	# 3	Compare
Multimodal Intent Recognition	PhotoChat	T5-base	F1	58.1	# 3	Compare
			Precision	58.2	# 2	Compare
			Recall	57.9	# 5	Compare
Multimodal Intent Recognition	PhotoChat	T5-3B	F1	58.9	# 2	Compare
			Precision	54.1	# 5	Compare
			Recall	64.6	# 2	Compare
Natural Language Inference	QNLI	T5-Base	Accuracy	93.7%	# 19	Compare
Natural Language Inference	QNLI	T5-Small	Accuracy	90.3%	# 35	Compare
Natural Language Inference	QNLI	T5-3B	Accuracy	96.3%	# 7	Compare
Natural Language Inference	QNLI	T5-11B	Accuracy	96.7%	# 6	Compare
Natural Language Inference	QNLI	T5-Large 770M	Accuracy	94.8%	# 12	Compare
Question Answering	Quora Question Pairs	T5-Large 770M	Accuracy	89.9%	# 9	Compare
Question Answering	Quora Question Pairs	T5-11B	Accuracy	90.4%	# 4	Compare
Question Answering	Quora Question Pairs	T5-Small	Accuracy	88.0%	# 16	Compare
Question Answering	Quora Question Pairs	T5-Base	Accuracy	89.4%	# 12	Compare
Question Answering	Quora Question Pairs	T5-3B	Accuracy	89.7%	# 11	Compare
Common Sense Reasoning	ReCoRD	T5-11B	F1	94.1	# 5	Compare
Common Sense Reasoning	ReCoRD	T5-XXL 11B (fine-tuned)	EM	93.4	# 6	Compare
Natural Language Inference	RTE	T5-Small	Accuracy	69.9%	# 54	Compare
Natural Language Inference	RTE	T5-XXL 11B (fine-tuned)	Accuracy	92.5%	# 8	Compare
Natural Language Inference	RTE	T5-Base 220M	Accuracy	80.1%	# 36	Compare
Natural Language Inference	RTE	T5-XL 3B	Accuracy	91.1%	# 14	Compare
Natural Language Inference	RTE	T5-Large 770M	Accuracy	87.2%	# 21	Compare
Question Answering	SQuAD1.1 dev	T5-Large 770M	EM	86.66	# 6	Compare
Question Answering	SQuAD1.1 dev	T5-Large 770M	F1	93.79	# 6	Compare
Question Answering	SQuAD1.1 dev	T5-Base	EM	85.44	# 8	Compare
Question Answering	SQuAD1.1 dev	T5-Base	F1	92.08	# 8	Compare
Question Answering	SQuAD1.1 dev	T5-3B	EM	88.53	# 5	Compare
Question Answering	SQuAD1.1 dev	T5-3B	F1	94.95	# 5	Compare
Question Answering	SQuAD1.1 dev	T5-Small	EM	79.1	# 16	Compare
Question Answering	SQuAD1.1 dev	T5-Small	F1	87.24	# 18	Compare
Question Answering	SQuAD1.1 dev	T5-11B	EM	90.06	# 1	Compare
Question Answering	SQuAD1.1 dev	T5-11B	F1	95.64	# 2	Compare
Sentiment Analysis	SST-2 Binary classification	T5-11B	Accuracy	97.5	# 1	Compare
Sentiment Analysis	SST-2 Binary classification	T5-Large 770M	Accuracy	96.3	# 17	Compare
Sentiment Analysis	SST-2 Binary classification	T5-3B	Accuracy	97.4	# 3	Compare
Sentiment Analysis	SST-2 Binary classification	T5-Base	Accuracy	95.2	# 24	Compare
Sentiment Analysis	SST-2 Binary classification	T5-Small	Accuracy	91.8	# 47	Compare
Semantic Textual Similarity	STS Benchmark	T5-11B	Pearson Correlation	0.925	# 4	Compare
Semantic Textual Similarity	STS Benchmark	T5-11B	Spearman Correlation	0.921	# 4	Compare
Semantic Textual Similarity	STS Benchmark	T5-Large 770M	Spearman Correlation	0.886	# 12	Compare
Semantic Textual Similarity	STS Benchmark	T5-Small	Pearson Correlation	0.856	# 25	Compare
Semantic Textual Similarity	STS Benchmark	T5-Small	Spearman Correlation	0.85	# 24	Compare
Semantic Textual Similarity	STS Benchmark	T5-Base	Pearson Correlation	0.894	# 22	Compare
Semantic Textual Similarity	STS Benchmark	T5-Large	Pearson Correlation	0.899	# 20	Compare
Semantic Textual Similarity	STS Benchmark	T5-3B	Pearson Correlation	0.906	# 17	Compare
Semantic Textual Similarity	STS Benchmark	T5-3B	Spearman Correlation	0.898	# 6	Compare
Question Answering	WebQuestions	T5.1.1-XXL+SSM	EM	42.8	# 6	Compare
Semantic Parsing	WebQuestionsSP	T5-11B (Raffel et al., 2020)	Accuracy	56.5	# 5	Compare
Poll Generation	WeiboPolls	T5	ROUGE-1	45.33	# 2	Compare
			ROUGE-L	42.69	# 2	Compare
			BLEU-1	37.34	# 2	Compare
			BLEU-3	21.06	# 2	Compare
Answer Generation	WeiboPolls	T5	ROUGE-1	46.20	# 2	Compare
			ROUGE-L	43.32	# 2	Compare
			BLEU-1	37.77	# 2	Compare
			BLEU-3	25.86	# 1	Compare
Question Generation	WeiboPolls	T5	ROUGE-1	44.46	# 2	Compare
			ROUGE-L	42.06	# 2	Compare
			BLEU-1	36.91	# 2	Compare
			BLEU-3	16.26	# 2	Compare
Coreference Resolution	Winograd Schema Challenge	T5-XXL 11B (fine-tuned)	Accuracy	93.8	# 7	Compare
Machine Translation	WMT2014 English-French	T5	BLEU score	43.4	# 9	Compare
Machine Translation	WMT2014 English-German	T5-11B	BLEU score	32.1	# 4	Compare
Machine Translation	WMT2014 English-German	T5-11B	Number of Params	11110M	# 1	Compare
Natural Language Inference	WNLI	T5-Base 220M	Accuracy	78.8	# 12	Compare
Natural Language Inference	WNLI	T5-Small 60M	Accuracy	69.2	# 18	Compare
Natural Language Inference	WNLI	T5-XXL 11B	Accuracy	93.2	# 3	Compare
Natural Language Inference	WNLI	T5-XL 3B	Accuracy	89.7	# 6	Compare
Natural Language Inference	WNLI	T5-Large 770M	Accuracy	85.6	# 10	Compare
Word Sense Disambiguation	Words in Context	T5-XXL 11B	Accuracy	76.9	# 8	Compare

Methods

Add Remove

Adafactor • Attention Dropout • BPE • Dense Connections • Dropout • GELU • GLU • Inverse Square Root Schedule • Layer Normalization • Linear Layer • Multi-Head Attention • Residual Connection • Scaled Dot-Product Attention • SentencePiece • Softmax • T5

Edit Social Preview

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods Edit Add Remove

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods

Add Remove