Augmenting Self-attention with Persistent Memory

2 Jul 2019  ·  Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lample, Herve Jegou, Armand Joulin ·

Transformer networks have lead to important progress in language modeling and machine translation. These models include two consecutive modules, a feed-forward layer and a self-attention layer. The latter allows the network to capture long term dependencies and are often regarded as the key ingredient in the success of Transformers. Building upon this intuition, we propose a new model that solely consists of attention layers. More precisely, we augment the self-attention layers with persistent memory vectors that play a similar role as the feed-forward layer. Thanks to these vectors, we can remove the feed-forward layer without degrading the performance of a transformer. Our evaluation shows the benefits brought by our model on standard character and word level language modeling benchmarks.

PDF Abstract


Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Language Modelling enwik8 All-attention network (36 layers) Number of params 114M # 10
Language Modelling enwik8 All-attention network (18 layers) Bit per Character (BPC) 1.01 # 18
Number of params 39M # 29
Language Modelling Text8 All-attention network - 18 layers Bit per Character (BPC) 1.11 # 7
Number of params 38M # 13
Language Modelling Text8 All-attention network - 36 layers Bit per Character (BPC) 1.08 # 4
Number of params 114M # 6
Language Modelling WikiText-103 All-attention network (36 layers) Validation perplexity 19.7 # 15
Test perplexity 20.6 # 25
Number of params 133M # 29