Transformers

Edit

Natural Language Processing • 38 methods

Methods

Add a Method

Method	Year	Papers
BERT BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding	2018	2113
Transformer Attention Is All You Need	2017	1905
RoBERTa RoBERTa: A Robustly Optimized BERT Pretraining Approach	2019	229
GPT-2 Language Models are Unsupervised Multitask Learners	2019	173
XLNet XLNet: Generalized Autoregressive Pretraining for Language Understanding	2019	92
ALBERT ALBERT: A Lite BERT for Self-supervised Learning of Language Representations	2019	67
BART BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension	2019	62
GPT Improving Language Understanding by Generative Pre-Training	2018	61
T5 Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer	2019	53
GPT-3 Language Models are Few-Shot Learners	2020	46
DistilBERT DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter	2019	34
Transformer-XL Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	2019	31
XLM Cross-lingual Language Model Pretraining	2019	31
ELECTRA ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators	2020	29
Longformer Longformer: The Long-Document Transformer	2020	17
ViLBERT ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks	2019	15
Sparse Transformer Generating Long Sequences with Sparse Transformers	2019	10
Reformer Reformer: The Efficient Transformer	2020	7
Universal Transformer Universal Transformers	2018	6
Levenshtein Transformer Levenshtein Transformer	2019	5
Linformer Linformer: Self-Attention with Linear Complexity	2020	5
CTRL CTRL: A Conditional Transformer Language Model for Controllable Generation	2019	4
VideoBERT VideoBERT: A Joint Model for Video and Language Representation Learning	2019	3
Adaptive Span Transformer Adaptive Attention Span in Transformers	2019	3
Routing Transformer Efficient Content-Based Sparse Attention with Routing Transformers	2020	2
TNT Transformer in Transformer	2021	2
Sinkhorn Transformer Sparse Sinkhorn Attention	2020	1
Sandwich Transformer Improving Transformer Models by Reordering their Sublayers	2019	1
BP-Transformer BP-Transformer: Modelling Long-Range Context via Binary Partitioning	2019	1
Compressive Transformer Compressive Transformers for Long-Range Sequence Modelling	2019	1
GPT-Neo	2000	1
T-D Generating Wikipedia by Summarizing Long Sequences	2018	1
Adaptively Sparse Transformer Adaptively Sparse Transformers	2019	1
Nyströmformer Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention	2021	1
DeLighT DeLighT: Deep and Light-weight Transformer	2020	1
GANsformer Generative Adversarial Transformers	2021	1
Feedback Transformer Addressing Some Limitations of Transformers with Feedback Memory	2020	1

Transformers Edit

Methods Add a Method

Transformers

Edit

Methods

Add a Method