Autoregressive Transformers

Edit

Methods

Add a Method

Method	Year	Papers
Transformer Attention Is All You Need	2017	4012
GPT-2 Language Models are Unsupervised Multitask Learners	2019	315
GPT-3 Language Models are Few-Shot Learners	2020	212
GPT Improving Language Understanding by Generative Pre-Training	2018	135
XLNet XLNet: Generalized Autoregressive Pretraining for Language Understanding	2019	132
Transformer-XL Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	2019	47
Sparse Transformer Generating Long Sequences with Sparse Transformers	2019	21
Universal Transformer Universal Transformers	2018	10
Linformer Linformer: Self-Attention with Linear Complexity	2020	10
Levenshtein Transformer Levenshtein Transformer	2019	8
Primer Primer: Searching for Efficient Transformers for Language Modeling	2021	6
Adaptive Span Transformer Adaptive Attention Span in Transformers	2019	3
Routing Transformer Efficient Content-Based Sparse Attention with Routing Transformers	2020	3
Sinkhorn Transformer Sparse Sinkhorn Attention	2020	1
DeLighT DeLighT: Deep and Light-weight Transformer	2020	1
Feedback Transformer Addressing Some Limitations of Transformers with Feedback Memory	2020	1
Sandwich Transformer Improving Transformer Models by Reordering their Sublayers	2019	1