Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence... (read more)

PDF Abstract ACL 2019 PDF ACL 2019 Abstract
TASK DATASET MODEL METRIC NAME METRIC VALUE GLOBAL RANK USES EXTRA
TRAINING DATA
RESULT BENCHMARK
Language Modelling enwik8 Transformer-XL (24 layers) Bit per Character (BPC) 0.99 # 7
Number of params 277M # 2
Language Modelling enwik8 Transformer-XL (18 layers) Bit per Character (BPC) 1.03 # 11
Number of params 88M # 10
Language Modelling enwik8 Transformer-XL (12 layers) Bit per Character (BPC) 1.06 # 12
Number of params 41M # 19
Language Modelling Hutter Prize 24-layer Transformer-XL Bit per Character (BPC) 0.99 # 4
Number of params 277M # 1
Language Modelling Hutter Prize 18-layer Transformer-XL Bit per Character (BPC) 1.03 # 6
Number of params 88M # 5
Language Modelling Hutter Prize 12-layer Transformer-XL Bit per Character (BPC) 1.06 # 7
Number of params 41M # 9
Language Modelling One Billion Word Transformer-XL Base PPL 23.5 # 6
Number of params 0.46B # 1
Language Modelling One Billion Word Transformer-XL Large PPL 21.8 # 3
Number of params 0.8B # 1
Language Modelling Penn Treebank (Word Level) Transformer-XL Validation perplexity 56.72 # 16
Test perplexity 54.55 # 20
Params 24M # 7
Language Modelling Text8 Transformer-XL - 24 layers Bit per Character (BPC) 1.08 # 4
Number of params 277M # 2
Language Modelling WikiText-103 Transformer-XL Large Validation perplexity 18.2 # 10
Test perplexity 18.3 # 13
Number of params 257M # 6
Language Modelling WikiText-103 Transformer-XL Standard Validation perplexity 23.1 # 17
Test perplexity 24.0 # 26
Number of params 151M # 11

Methods used in the Paper


METHOD TYPE
Cosine Annealing
Learning Rate Schedules
Softmax
Output Functions
Multi-Head Attention
Attention Modules
Residual Connection
Skip Connections
Adaptive Input Representations
Input Embedding Factorization
Adaptive Softmax
Output Functions
Linear Warmup With Cosine Annealing
Learning Rate Schedules
Dense Connections
Feedforward Networks
ReLU
Activation Functions
Variational Dropout
Regularization
Adam
Stochastic Optimization
Dropout
Regularization
Layer Normalization
Normalization
Scaled Dot-Product Attention
Attention Mechanisms
Transformer-XL
Transformers