Autoregressive Transformers

Edit

Methods

Add a Method

Method	Year	Papers
Transformer Attention Is All You Need	2017	6919
GPT-3 Language Models are Few-Shot Learners	2020	830
GPT-2 Language Models are Unsupervised Multitask Learners	2019	463
GPT Improving Language Understanding by Generative Pre-Training	2018	378
XLNet XLNet: Generalized Autoregressive Pretraining for Language Understanding	2019	149
Transformer-XL Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	2019	55
Sparse Transformer Generating Long Sequences with Sparse Transformers	2019	31
Universal Transformer Universal Transformers	2018	14
Linformer Linformer: Self-Attention with Linear Complexity	2020	13
Levenshtein Transformer Levenshtein Transformer	2019	10
Primer Primer: Searching for Efficient Transformers for Language Modeling	2021	9
Adaptive Span Transformer Adaptive Attention Span in Transformers	2019	3
Routing Transformer Efficient Content-Based Sparse Attention with Routing Transformers	2020	3
Sinkhorn Transformer Sparse Sinkhorn Attention	2020	1
DeLighT DeLighT: Deep and Light-weight Transformer	2020	1
Feedback Transformer Addressing Some Limitations of Transformers with Feedback Memory	2020	1
Sandwich Transformer Improving Transformer Models by Reordering their Sublayers	2019	1