Autoregressive Transformers

Edit

Methods

Add a Method

Method	Year	Papers
Transformer Attention Is All You Need	2017	9338
GPT-3 Language Models are Few-Shot Learners	2020	1324
GPT Improving Language Understanding by Generative Pre-Training	2018	654
GPT-2 Language Models are Unsupervised Multitask Learners	2019	543
XLNet XLNet: Generalized Autoregressive Pretraining for Language Understanding	2019	156
Transformer-XL Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	2019	61
Sparse Transformer Generating Long Sequences with Sparse Transformers	2019	35
Universal Transformer Universal Transformers	2018	17
Linformer Linformer: Self-Attention with Linear Complexity	2020	13
Levenshtein Transformer Levenshtein Transformer	2019	11
Primer Primer: Searching for Efficient Transformers for Language Modeling	2021	10
Adaptive Span Transformer Adaptive Attention Span in Transformers	2019	3
Routing Transformer Efficient Content-Based Sparse Attention with Routing Transformers	2020	3
Feedback Transformer Addressing Some Limitations of Transformers with Feedback Memory	2020	2
Sinkhorn Transformer Sparse Sinkhorn Attention	2020	1
DeLighT DeLighT: Deep and Light-weight Transformer	2020	1
Sandwich Transformer Improving Transformer Models by Reordering their Sublayers	2019	1