TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Common Sense Reasoning	ARC (Challenge)	ST-MoE-L 4.1B (fine-tuned)	Accuracy	56.9	# 28
Common Sense Reasoning	ARC (Challenge)	ST-MoE-32B 269B (fine-tuned)	Accuracy	86.5	# 12
Common Sense Reasoning	ARC (Easy)	ST-MoE-32B 269B (fine-tuned)	Accuracy	95.2	# 1
Common Sense Reasoning	ARC (Easy)	ST-MoE-L 4.1B (fine-tuned)	Accuracy	75.4	# 23
Question Answering	BoolQ	ST-MoE-32B 269B (fine-tuned)	Accuracy	92.4	# 2
Question Answering	BoolQ	ST-MoE-L 4.1B (fine-tuned)	Accuracy	88.6	# 10
Natural Language Inference	CommitmentBank	ST-MoE-32B 269B (fine-tuned)	Accuracy	98	# 4
Natural Language Inference	CommitmentBank	ST-MoE-L 4.1B (fine-tuned)	Accuracy	98.2	# 3
Question Answering	COPA	ST-MoE-L 4.1B (fine-tuned)	Accuracy	91	# 13
Question Answering	COPA	ST-MoE-32B 269B (fine-tuned)	Accuracy	99.2	# 3
Question Answering	MultiRC	ST-MoE-32B 269B (fine-tuned)	F1	89.6	# 2
Question Answering	MultiRC	ST-MoE-L 4.1B (fine-tuned)	F1	86	# 8
Common Sense Reasoning	ReCoRD	ST-MoE-32B 269B (fine-tuned)	EM	95.1	# 2
Common Sense Reasoning	ReCoRD	ST-MoE-L 4.1B (fine-tuned)	EM	88.9	# 12
Natural Language Inference	RTE	ST-MoE-L 4.1B (fine-tuned)	Accuracy	92.1%	# 10
Natural Language Inference	RTE	ST-MoE-32B 269B (fine-tuned)	Accuracy	93.5%	# 4
Coreference Resolution	Winograd Schema Challenge	ST-MoE-L 4.1B (fine-tuned)	Accuracy	93.3	# 8
Coreference Resolution	Winograd Schema Challenge	ST-MoE-32B 269B (fine-tuned)	Accuracy	96.6	# 5
Common Sense Reasoning	WinoGrande	ST-MoE-32B 269B (fine-tuned)	Accuracy	96.1	# 1
Common Sense Reasoning	WinoGrande	ST-MoE-L 4.1B (fine-tuned)	Accuracy	81.7	# 13
Word Sense Disambiguation	Words in Context	ST-MoE-32B 269B (fine-tuned)	Accuracy	77.7	# 3
Word Sense Disambiguation	Words in Context	ST-MoE-L 4.1B (fine-tuned)	Accuracy	74	# 10

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/designing-effective-sparse-expert-models/common-sense-reasoning-on-arc-easy)](https://paperswithcode.com/sota/common-sense-reasoning-on-arc-easy?p=designing-effective-sparse-expert-models)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/designing-effective-sparse-expert-models/common-sense-reasoning-on-winogrande)](https://paperswithcode.com/sota/common-sense-reasoning-on-winogrande?p=designing-effective-sparse-expert-models)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/designing-effective-sparse-expert-models/question-answering-on-boolq)](https://paperswithcode.com/sota/question-answering-on-boolq?p=designing-effective-sparse-expert-models)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/designing-effective-sparse-expert-models/question-answering-on-multirc)](https://paperswithcode.com/sota/question-answering-on-multirc?p=designing-effective-sparse-expert-models)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/designing-effective-sparse-expert-models/common-sense-reasoning-on-record)](https://paperswithcode.com/sota/common-sense-reasoning-on-record?p=designing-effective-sparse-expert-models)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/designing-effective-sparse-expert-models/natural-language-inference-on-commitmentbank)](https://paperswithcode.com/sota/natural-language-inference-on-commitmentbank?p=designing-effective-sparse-expert-models)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/designing-effective-sparse-expert-models/question-answering-on-copa)](https://paperswithcode.com/sota/question-answering-on-copa?p=designing-effective-sparse-expert-models)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/designing-effective-sparse-expert-models/word-sense-disambiguation-on-words-in-context)](https://paperswithcode.com/sota/word-sense-disambiguation-on-words-in-context?p=designing-effective-sparse-expert-models)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/designing-effective-sparse-expert-models/natural-language-inference-on-rte)](https://paperswithcode.com/sota/natural-language-inference-on-rte?p=designing-effective-sparse-expert-models)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/designing-effective-sparse-expert-models/coreference-resolution-on-winograd-schema)](https://paperswithcode.com/sota/coreference-resolution-on-winograd-schema?p=designing-effective-sparse-expert-models)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/designing-effective-sparse-expert-models/common-sense-reasoning-on-arc-challenge)](https://paperswithcode.com/sota/common-sense-reasoning-on-arc-challenge?p=designing-effective-sparse-expert-models)`

ST-MoE: Designing Stable and Transferable Sparse Expert Models

17 Feb 2022 · Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, William Fedus ·

Scale has opened new frontiers in natural language processing -- but at a high cost. In response, Mixture-of-Experts (MoE) and Switch Transformers have been proposed as an energy efficient path to even larger and more capable language models. But advancing the state-of-the-art across a broad set of natural language tasks has been hindered by training instabilities and uncertain quality during fine-tuning. Our work focuses on these issues and acts as a design guide. We conclude by scaling a sparse model to 269B parameters, with a computational cost comparable to a 32B dense encoder-decoder Transformer (Stable and Transferable Mixture-of-Experts or ST-MoE-32B). For the first time, a sparse model achieves state-of-the-art performance in transfer learning, across a diverse set of tasks including reasoning (SuperGLUE, ARC Easy, ARC Challenge), summarization (XSum, CNN-DM), closed book question answering (WebQA, Natural Questions), and adversarially constructed tasks (Winogrande, ANLI R3).

PDF Abstract

Code

Add Remove Mark official

tensorflow/mesh official

1,606

xuefuzhao/openmoe

↳ Quickstart in

Colab

1,527

yikangshen/megablocks

Tasks

Add Remove

ARC

Common Sense Reasoning

Coreference Resolution

Decoder

Mixture-of-Experts

Natural Language Inference

Natural Questions

Question Answering

ST-MoE-L 4.1B (fine-tuned)

Transfer Learning

Winogrande

Word Sense Disambiguation

Datasets

GLUE

Natural Questions

TriviaQA

WinoGrande

BoolQ

SuperGLUE

WSC

COPA

ANLI

ARC (AI2 Reasoning Challenge)

MultiRC mC4

ReCoRD RTE CommitmentBank

Results from the Paper

Edit

Ranked #1 on Common Sense Reasoning on ARC (Easy)

Get a GitHub badge

Task	Dataset	Model	Metric Name	Metric Value	Global Rank	Benchmark
Common Sense Reasoning	ARC (Challenge)	ST-MoE-L 4.1B (fine-tuned)	Accuracy	56.9	# 28	Compare
Common Sense Reasoning	ARC (Challenge)	ST-MoE-32B 269B (fine-tuned)	Accuracy	86.5	# 12	Compare
Common Sense Reasoning	ARC (Easy)	ST-MoE-32B 269B (fine-tuned)	Accuracy	95.2	# 1	Compare
Common Sense Reasoning	ARC (Easy)	ST-MoE-L 4.1B (fine-tuned)	Accuracy	75.4	# 23	Compare
Question Answering	BoolQ	ST-MoE-32B 269B (fine-tuned)	Accuracy	92.4	# 2	Compare
Question Answering	BoolQ	ST-MoE-L 4.1B (fine-tuned)	Accuracy	88.6	# 10	Compare
Natural Language Inference	CommitmentBank	ST-MoE-32B 269B (fine-tuned)	Accuracy	98	# 4	Compare
Natural Language Inference	CommitmentBank	ST-MoE-L 4.1B (fine-tuned)	Accuracy	98.2	# 3	Compare
Question Answering	COPA	ST-MoE-L 4.1B (fine-tuned)	Accuracy	91	# 13	Compare
Question Answering	COPA	ST-MoE-32B 269B (fine-tuned)	Accuracy	99.2	# 3	Compare
Question Answering	MultiRC	ST-MoE-32B 269B (fine-tuned)	F1	89.6	# 2	Compare
Question Answering	MultiRC	ST-MoE-L 4.1B (fine-tuned)	F1	86	# 8	Compare
Common Sense Reasoning	ReCoRD	ST-MoE-32B 269B (fine-tuned)	EM	95.1	# 2	Compare
Common Sense Reasoning	ReCoRD	ST-MoE-L 4.1B (fine-tuned)	EM	88.9	# 12	Compare
Natural Language Inference	RTE	ST-MoE-L 4.1B (fine-tuned)	Accuracy	92.1%	# 10	Compare
Natural Language Inference	RTE	ST-MoE-32B 269B (fine-tuned)	Accuracy	93.5%	# 4	Compare
Coreference Resolution	Winograd Schema Challenge	ST-MoE-L 4.1B (fine-tuned)	Accuracy	93.3	# 8	Compare
Coreference Resolution	Winograd Schema Challenge	ST-MoE-32B 269B (fine-tuned)	Accuracy	96.6	# 5	Compare
Common Sense Reasoning	WinoGrande	ST-MoE-32B 269B (fine-tuned)	Accuracy	96.1	# 1	Compare
Common Sense Reasoning	WinoGrande	ST-MoE-L 4.1B (fine-tuned)	Accuracy	81.7	# 13	Compare
Word Sense Disambiguation	Words in Context	ST-MoE-32B 269B (fine-tuned)	Accuracy	77.7	# 3	Compare
Word Sense Disambiguation	Words in Context	ST-MoE-L 4.1B (fine-tuned)	Accuracy	74	# 10	Compare

Methods

Add Remove

Absolute Position Encodings • Adam • BPE • Dense Connections • Dropout • Label Smoothing • Layer Normalization • Linear Layer • MoE • Multi-Head Attention • Position-Wise Feed-Forward Layer • Residual Connection • Scaled Dot-Product Attention • Softmax • Transformer

Edit Social Preview

ST-MoE: Designing Stable and Transferable Sparse Expert Models

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods Edit Add Remove

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods

Add Remove