TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Language Modelling	enwik8	Feedback Transformer	Bit per Character (BPC)	0.96	# 3
Language Modelling	enwik8	Feedback Transformer	Number of params	77M	# 9
Language Modelling	Penn Treebank (Character Level)	Feedback Transformer	Bit per Character (BPC)	1.160	# 3
Language Modelling	Penn Treebank (Character Level)	Feedback Transformer	Number of params	10.7M	# 7
Language Modelling	WikiText-103	Feedback Transformer (4 layers)	Validation perplexity	21.4	# 13
Language Modelling	WikiText-103	Feedback Transformer (4 layers)	Test perplexity	22.4	# 20
Language Modelling	WikiText-103	Feedback Transformer (4 layers)	Number of params	44M	# 17
Language Modelling	WikiText-103	Feedback Transformer (8 layers)	Validation perplexity	17.5	# 7
Language Modelling	WikiText-103	Feedback Transformer (8 layers)	Test perplexity	18.2	# 12
Language Modelling	WikiText-103	Feedback Transformer (8 layers)	Number of params	139M	# 10

🦡 Badges

Include the markdown at the top of your GitHub README.md file to showcase the performance of the model.

Badges are live and will be dynamically updated with the latest ranking of this paper.

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/accessing-higher-level-representations-in/language-modelling-on-enwiki8)](https://paperswithcode.com/sota/language-modelling-on-enwiki8?p=accessing-higher-level-representations-in)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/accessing-higher-level-representations-in/language-modelling-on-penn-treebank-character)](https://paperswithcode.com/sota/language-modelling-on-penn-treebank-character?p=accessing-higher-level-representations-in)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/accessing-higher-level-representations-in/language-modelling-on-wikitext-103)](https://paperswithcode.com/sota/language-modelling-on-wikitext-103?p=accessing-higher-level-representations-in)`

Addressing Some Limitations of Transformers with Feedback Memory

21 Feb 2020 • Angela Fan • Thibaut Lavril • Edouard Grave • Armand Joulin • Sainbayar Sukhbaatar

Transformers have been successfully applied to sequential, auto-regressive tasks despite being feedforward networks. Unlike recurrent neural networks, Transformers use attention to capture temporal relations while processing input tokens in parallel... (read more)

PDF Abstract PDF Abstract

Code

Add Remove Mark official

lab-ml/nn

670

Tasks

Add Remove

LANGUAGE MODELLING

MACHINE TRANSLATION

Results from the Paper

Edit

Ranked #3 on Language Modelling on enwik8

Get a GitHub badge

TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK	BENCHMARK
Language Modelling	enwik8	Feedback Transformer	Bit per Character (BPC)	0.96	# 3	Compare
Language Modelling	enwik8	Feedback Transformer	Number of params	77M	# 9	Compare
Language Modelling	Penn Treebank (Character Level)	Feedback Transformer	Bit per Character (BPC)	1.160	# 3	Compare
Language Modelling	Penn Treebank (Character Level)	Feedback Transformer	Number of params	10.7M	# 7	Compare
Language Modelling	WikiText-103	Feedback Transformer (4 layers)	Validation perplexity	21.4	# 13	Compare
			Test perplexity	22.4	# 20	Compare
			Number of params	44M	# 17	Compare
Language Modelling	WikiText-103	Feedback Transformer (8 layers)	Validation perplexity	17.5	# 7	Compare
			Test perplexity	18.2	# 12	Compare
			Number of params	139M	# 10	Compare

Methods used in the Paper

Add Remove

METHOD	TYPE
GELU	Activation Functions
Feedback Memory	Attention Modules
Tanh Activation	Activation Functions
Sigmoid Activation	Activation Functions
LSTM	Recurrent Neural Networks
A2C	Policy Gradient Methods
RMSProp	Stochastic Optimization
Residual Connection	Skip Connections
Label Smoothing	Regularization
Dropout	Regularization
BPE	Subword Segmentation
Adam	Stochastic Optimization
Dense Connections	Feedforward Networks
Multi-Head Attention	Attention Modules
Layer Normalization	Normalization
Scaled Dot-Product Attention	Attention Mechanisms
Feedback Transformer	Transformers
Transformer	Transformers
Softmax	Output Functions

Addressing Some Limitations of Transformers with Feedback Memory

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Results from the Paper Edit

Methods used in the Paper Edit Add Remove

Code

Add Remove Mark official

Tasks

Add Remove

Results from the Paper

Edit

Methods used in the Paper

Add Remove