Compressive Transformers for Long-Range Sequence Modelling
We present the Compressive Transformer, an attentive sequence model which compresses past memories for long-range sequence learning. We find the Compressive Transformer obtains state-of-the-art language modelling results in the WikiText-103 and Enwik8 benchmarks, achieving 17.1 ppl and 0.97 bpc respectively. We also find it can model high-frequency speech effectively and can be used as a memory mechanism for RL, demonstrated on an object matching task. To promote the domain of long-range sequence learning, we propose a new open-vocabulary language modelling benchmark derived from books, PG-19.
PDF Abstract ICLR 2020 PDF ICLR 2020 AbstractCode
Tasks
Datasets
Introduced in the Paper:
PG-19Used in the Paper:
WikiText-2
WikiText-103
BookCorpus
LAMBADA
NarrativeQA
Billion Word Benchmark
CBT
Hutter Prize
Methods
Adam •
Adaptive Input Representations •
Adaptive Softmax •
Compressed Memory •
Compressive Transformer •
Cosine Annealing •
Dense Connections •
Dropout •
Gradient Clipping •
Layer Normalization •
Linear Layer •
Linear Warmup With Cosine Annealing •
Multi-Head Attention •
ReLU •
Residual Connection •
Scaled Dot-Product Attention •
Softmax •
Transformer-XL •
Variational Dropout
