Transformers

Edit

Natural Language Processing • Language Models • 86 methods

Transformers are a type of neural network architecture that have several properties that make them effective for modeling data with long-range dependencies. They generally feature a combination of multi-headed attention mechanisms, residual connections, layer normalization, feedforward connections, and positional embeddings.

Subcategories

1 Autoencoding Transformers

2 Autoregressive Transformers

3 Rendezvous

Methods

Add a Method

Method	Year	Papers
Transformer Attention Is All You Need	2017	3767
BERT BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding	2018	3539
RoBERTa RoBERTa: A Robustly Optimized BERT Pretraining Approach	2019	452
GPT-2 Language Models are Unsupervised Multitask Learners	2019	307
T5 Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer	2019	203
GPT-3 Language Models are Few-Shot Learners	2020	195
BART BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension	2019	180
GPT Improving Language Understanding by Generative Pre-Training	2018	133
XLNet XLNet: Generalized Autoregressive Pretraining for Language Understanding	2019	131
ALBERT ALBERT: A Lite BERT for Self-supervised Learning of Language Representations	2019	126
DistilBERT DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter	2019	73
ELECTRA ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators	2020	67
XLM Cross-lingual Language Model Pretraining	2019	46
Transformer-XL Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	2019	45
Electric Pre-Training Transformers as Energy-Based Cloze Models	2020	41
Performer Rethinking Attention with Performers	2020	35
Longformer Longformer: The Long-Document Transformer	2020	33
mBART Multilingual Denoising Pre-training for Neural Machine Translation	2020	31
mT5 mT5: A massively multilingual pre-trained text-to-text transformer	2020	29
CodeBERT CodeBERT: A Pre-Trained Model for Programming and Natural Languages	2020	22
ViLBERT ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks	2019	21
ETC ETC: Encoding Long and Structured Inputs in Transformers	2020	19
Sparse Transformer Generating Long Sequences with Sparse Transformers	2019	18
DeBERTa DeBERTa: Decoding-enhanced BERT with Disentangled Attention	2020	13
Reformer Reformer: The Efficient Transformer	2020	12
Switch Transformer Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity	2021	11
Universal Transformer Universal Transformers	2018	10
Linformer Linformer: Self-Attention with Linear Complexity	2020	10
RAG Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks	2020	10
BigBird Big Bird: Transformers for Longer Sequences	2020	9
GPT-Neo	2000	8
Levenshtein Transformer Levenshtein Transformer	2019	8
CTRL CTRL: A Conditional Transformer Language Model for Controllable Generation	2019	7
ProphetNet ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training	2020	6
TNT Transformer in Transformer	2021	6
CodeT5 CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation	2021	5
Primer Primer: Searching for Efficient Transformers for Language Modeling	2021	5
PLATO-2 PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learning	2020	4
Charformer Charformer: Fast Character Transformers via Gradient-based Subword Tokenization	2021	4
Bort Optimal Subarchitecture Extraction For BERT	2020	4
MobileBERT MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices	2020	4
Routing Transformer Efficient Content-Based Sparse Attention with Routing Transformers	2020	3
Adaptive Span Transformer Adaptive Attention Span in Transformers	2019	3
VideoBERT VideoBERT: A Joint Model for Video and Language Representation Learning	2019	3
EGT Global Self-Attention as a Replacement for Graph Convolution	2021	3
Compressive Transformer Compressive Transformers for Long-Range Sequence Modelling	2019	2
DeeBERT DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference	2020	2
CuBERT Learning and Evaluating Contextual Embedding of Source Code	2019	2
Subformer Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers	2021	2
DynaBERT DynaBERT: Dynamic BERT with Adaptive Width and Depth	2020	2
TernaryBERT TernaryBERT: Distillation-aware Ultra-low Bit BERT	2020	2
Fastformer Fastformer: Additive Attention Can Be All You Need	2021	2
MATE MATE: Multi-view Attention for Table Transformer Efficiency	2021	2
Nyströmformer Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention	2021	2
PermuteFormer PermuteFormer: Efficient Relative Position Encoding for Long Sequences	2021	1
NormFormer NormFormer: Improved Transformer Pretraining with Extra Normalization	2021	1
BP-Transformer BP-Transformer: Modelling Long-Range Context via Binary Partitioning	2019	1
IB-BERT MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices	2020	1
MacBERT Revisiting Pre-Trained Models for Chinese Natural Language Processing	2020	1
SMITH Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching	2020	1
ClipBERT Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling	2021	1
I-BERT I-BERT: Integer-only BERT Quantization	2021	1
SqueezeBERT SqueezeBERT: What can computer vision teach NLP about efficient neural networks?	2020	1
Sandwich Transformer Improving Transformer Models by Reordering their Sublayers	2019	1
DeLighT DeLighT: Deep and Light-weight Transformer	2020	1
PAR Transformer Pay Attention when Required	2020	1
ConvBERT ConvBERT: Improving BERT with Span-based Dynamic Convolution	2020	1
ESACL Enhanced Seq2Seq Autoencoder via Contrastive Learning for Abstractive Text Summarization	2021	1
AutoTinyBERT AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models	2021	1
MHMA Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos	2021	1
RealFormer RealFormer: Transformer Likes Residual Attention	2020	1
Sinkhorn Transformer Sparse Sinkhorn Attention	2020	1
SongNet SongNet: Rigid Formats Controlled Text Generation	2020	1
Funnel Transformer Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing	2020	1
T-D Generating Wikipedia by Summarizing Long Sequences	2018	1
SC-GPT Few-shot Natural Language Generation for Task-Oriented Dialog	2020	1
Chinese Pre-trained Unbalanced Transformer CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation	2021	1
GANformer Generative Adversarial Transformers	2021	1
BinaryBERT BinaryBERT: Pushing the Limit of BERT Quantization	2020	1
Adaptively Sparse Transformer Adaptively Sparse Transformers	2019	1
Feedback Transformer Addressing Some Limitations of Transformers with Feedback Memory	2020	1

Transformers Edit

Methods Add a Method

Transformers

Edit

Methods

Add a Method