TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Question Answering	ARC-c	FLAN 137B zero-shot	Accuracy	63.1	# 1
Question Answering	ARC-e	FLAN 137B zero-shot	Accuracy	79.6	# 1
Question Answering	BoolQ	FLAN 137B (zero-shot)	Accuracy	82.9	# 17
Question Answering	COPA	FLAN 137B zero-shot	Accuracy	91.0	# 7
Sentence Completion	HellaSwag	FLAN 137B (zero-shot)	Accuracy	56.7	# 36
Question Answering	MultiRC	FLAN 137B (zero-shot)	F1	77.5	# 7
Question Answering	NaturalQA	FLAN 137B zero-shot	EM	20.7	# 2
Question Answering	OBQA	FLAN 137B zero-shot	Accuracy	78.4	# 1
Question Answering	PIQA	FLAN 137B (zero-shot)	Accuracy	80.5	# 15
Common Sense Reasoning	ReCoRD	FLAN 137B zero-shot	F1	72.5	# 23
Natural Language Inference	RTE	FLAN 137B zero-shot	Accuracy	84.1%	# 17
Question Answering	Story Cloze	FLAN 137B zero-shot	Accuracy	93.4	# 1
Question Answering	TriviaQA	FLAN 137B zero-shot	EM	56.7	# 27
Machine Translation	WMT2014 English-French	FLAN 137B zero-shot	BLEU score	34	# 48
Machine Translation	WMT2014 French-English	FLAN 137B zero-shot	BLEU score	36.5	# 1
Machine Translation	WMT2016 English-German	FLAN 137B zero-shot	BLEU score	27.0	# 5
Machine Translation	WMT2016 English-Romanian	FLAN 137B zero-shot	BLEU score	18.4	# 19
Machine Translation	WMT2016 German-English	FLAN 137B zero-shot	BLEU score	39.8	# 1
Machine Translation	WMT2016 Romanian-English	FLAN 137B zero-shot	BLEU score	36.7	# 2
Common Sense Reasoning	WSC273	FLAN 137B zero-shot	Accuracy	80.8	# 1

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/finetuned-language-models-are-zero-shot/question-answering-on-arc-c)](https://paperswithcode.com/sota/question-answering-on-arc-c?p=finetuned-language-models-are-zero-shot)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/finetuned-language-models-are-zero-shot/question-answering-on-arc-e)](https://paperswithcode.com/sota/question-answering-on-arc-e?p=finetuned-language-models-are-zero-shot)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/finetuned-language-models-are-zero-shot/question-answering-on-obqa)](https://paperswithcode.com/sota/question-answering-on-obqa?p=finetuned-language-models-are-zero-shot)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/finetuned-language-models-are-zero-shot/question-answering-on-storycloze)](https://paperswithcode.com/sota/question-answering-on-storycloze?p=finetuned-language-models-are-zero-shot)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/finetuned-language-models-are-zero-shot/machine-translation-on-wmt2014-french-english)](https://paperswithcode.com/sota/machine-translation-on-wmt2014-french-english?p=finetuned-language-models-are-zero-shot)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/finetuned-language-models-are-zero-shot/machine-translation-on-wmt2016-german-english)](https://paperswithcode.com/sota/machine-translation-on-wmt2016-german-english?p=finetuned-language-models-are-zero-shot)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/finetuned-language-models-are-zero-shot/common-sense-reasoning-on-wsc273)](https://paperswithcode.com/sota/common-sense-reasoning-on-wsc273?p=finetuned-language-models-are-zero-shot)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/finetuned-language-models-are-zero-shot/question-answering-on-naturalqa)](https://paperswithcode.com/sota/question-answering-on-naturalqa?p=finetuned-language-models-are-zero-shot)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/finetuned-language-models-are-zero-shot/machine-translation-on-wmt2016-romanian)](https://paperswithcode.com/sota/machine-translation-on-wmt2016-romanian?p=finetuned-language-models-are-zero-shot)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/finetuned-language-models-are-zero-shot/machine-translation-on-wmt2016-english-german)](https://paperswithcode.com/sota/machine-translation-on-wmt2016-english-german?p=finetuned-language-models-are-zero-shot)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/finetuned-language-models-are-zero-shot/question-answering-on-copa)](https://paperswithcode.com/sota/question-answering-on-copa?p=finetuned-language-models-are-zero-shot)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/finetuned-language-models-are-zero-shot/question-answering-on-multirc)](https://paperswithcode.com/sota/question-answering-on-multirc?p=finetuned-language-models-are-zero-shot)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/finetuned-language-models-are-zero-shot/question-answering-on-piqa)](https://paperswithcode.com/sota/question-answering-on-piqa?p=finetuned-language-models-are-zero-shot)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/finetuned-language-models-are-zero-shot/question-answering-on-boolq)](https://paperswithcode.com/sota/question-answering-on-boolq?p=finetuned-language-models-are-zero-shot)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/finetuned-language-models-are-zero-shot/natural-language-inference-on-rte)](https://paperswithcode.com/sota/natural-language-inference-on-rte?p=finetuned-language-models-are-zero-shot)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/finetuned-language-models-are-zero-shot/machine-translation-on-wmt2016-english-1)](https://paperswithcode.com/sota/machine-translation-on-wmt2016-english-1?p=finetuned-language-models-are-zero-shot)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/finetuned-language-models-are-zero-shot/common-sense-reasoning-on-record)](https://paperswithcode.com/sota/common-sense-reasoning-on-record?p=finetuned-language-models-are-zero-shot)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/finetuned-language-models-are-zero-shot/question-answering-on-triviaqa)](https://paperswithcode.com/sota/question-answering-on-triviaqa?p=finetuned-language-models-are-zero-shot)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/finetuned-language-models-are-zero-shot/sentence-completion-on-hellaswag)](https://paperswithcode.com/sota/sentence-completion-on-hellaswag?p=finetuned-language-models-are-zero-shot)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/finetuned-language-models-are-zero-shot/machine-translation-on-wmt2014-english-french)](https://paperswithcode.com/sota/machine-translation-on-wmt2014-english-french?p=finetuned-language-models-are-zero-shot)`

Finetuned Language Models Are Zero-Shot Learners

ICLR 2022 · Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, Quoc V. Le ·

This paper explores a simple method for improving the zero-shot learning abilities of language models. We show that instruction tuning -- finetuning language models on a collection of tasks described via instructions -- substantially improves zero-shot performance on unseen tasks. We take a 137B parameter pretrained language model and instruction-tune it on over 60 NLP tasks verbalized via natural language instruction templates. We evaluate this instruction-tuned model, which we call FLAN, on unseen task types. FLAN substantially improves the performance of its unmodified counterpart and surpasses zero-shot 175B GPT-3 on 20 of 25 tasks that we evaluate. FLAN even outperforms few-shot GPT-3 by a large margin on ANLI, RTE, BoolQ, AI2-ARC, OpenbookQA, and StoryCloze. Ablation studies reveal that number of finetuning datasets, model scale, and natural language instructions are key to the success of instruction tuning.

PDF Abstract ICLR 2022 PDF ICLR 2022 Abstract

Code

Add Remove Mark official

google-research/flan official

1,277

hiyouga/llama-efficient-tuning

↳ Quickstart in

Spaces

7,641

bigcode-project/starcoder

↳ Quickstart in

Spaces

6,682

bigscience-workshop/promptsource

↳ Quickstart in

Spaces

2,274

openbiolink/promptsource

Tasks

Add Remove

Common Sense Reasoning

Language Modelling

Machine Translation

Natural Language Inference

Question Answering

RTE

Sentence Completion

Zero-Shot Learning

Datasets

GLUE

SNLI

QNLI

Natural Questions

SuperGLUE

TriviaQA

BoolQ

WMT 2014

DROP

COPA

HellaSwag

PIQA

ANLI

ARC

OpenBookQA

WMT 2016

MultiRC

ReCoRD ParaCrawl Story Cloze RTE

WMT 2016 News

Results from the Paper

Edit

Ranked #1 on Question Answering on Story Cloze

Get a GitHub badge

Task	Dataset	Model	Metric Name	Metric Value	Global Rank	Benchmark
Question Answering	ARC-c	FLAN 137B zero-shot	Accuracy	63.1	# 1	Compare
Question Answering	ARC-e	FLAN 137B zero-shot	Accuracy	79.6	# 1	Compare
Question Answering	BoolQ	FLAN 137B (zero-shot)	Accuracy	82.9	# 17	Compare
Question Answering	COPA	FLAN 137B zero-shot	Accuracy	91.0	# 7	Compare
Sentence Completion	HellaSwag	FLAN 137B (zero-shot)	Accuracy	56.7	# 36	Compare
Question Answering	MultiRC	FLAN 137B (zero-shot)	F1	77.5	# 7	Compare
Question Answering	NaturalQA	FLAN 137B zero-shot	EM	20.7	# 2	Compare
Question Answering	OBQA	FLAN 137B zero-shot	Accuracy	78.4	# 1	Compare
Question Answering	PIQA	FLAN 137B (zero-shot)	Accuracy	80.5	# 15	Compare
Common Sense Reasoning	ReCoRD	FLAN 137B zero-shot	F1	72.5	# 23	Compare
Natural Language Inference	RTE	FLAN 137B zero-shot	Accuracy	84.1%	# 17	Compare
Question Answering	Story Cloze	FLAN 137B zero-shot	Accuracy	93.4	# 1	Compare
Question Answering	TriviaQA	FLAN 137B zero-shot	EM	56.7	# 27	Compare
Machine Translation	WMT2014 English-French	FLAN 137B zero-shot	BLEU score	34	# 48	Compare
Machine Translation	WMT2014 French-English	FLAN 137B zero-shot	BLEU score	36.5	# 1	Compare
Machine Translation	WMT2016 English-German	FLAN 137B zero-shot	BLEU score	27.0	# 5	Compare
Machine Translation	WMT2016 English-Romanian	FLAN 137B zero-shot	BLEU score	18.4	# 19	Compare
Machine Translation	WMT2016 German-English	FLAN 137B zero-shot	BLEU score	39.8	# 1	Compare
Machine Translation	WMT2016 Romanian-English	FLAN 137B zero-shot	BLEU score	36.7	# 2	Compare
Common Sense Reasoning	WSC273	FLAN 137B zero-shot	Accuracy	80.8	# 1	Compare

Methods

Add Remove

Adam • Attention Dropout • BPE • Cosine Annealing • Dense Connections • Dropout • Fixed Factorized Attention • GELU • GPT-3 • Layer Normalization • Linear Layer • Linear Warmup With Cosine Annealing • Multi-Head Attention • Residual Connection • Scaled Dot-Product Attention • Softmax • Strided Attention • Weight Decay

Edit Social Preview

Finetuned Language Models Are Zero-Shot Learners

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods Edit Add Remove

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods

Add Remove