Transformers

Edit

Natural Language Processing • Language Models • 92 methods

Transformers are a type of neural network architecture that have several properties that make them effective for modeling data with long-range dependencies. They generally feature a combination of multi-headed attention mechanisms, residual connections, layer normalization, feedforward connections, and positional embeddings.

Subcategories

1 Autoencoding Transformers

2 Autoregressive Transformers

3 Rendezvous

Methods

Add a Method

Method	Year	Papers
Transformer Attention Is All You Need	2017	5884
BERT BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding	2018	4237
GPT-3 Language Models are Few-Shot Learners	2020	606
RoBERTa RoBERTa: A Robustly Optimized BERT Pretraining Approach	2019	591
GPT-2 Language Models are Unsupervised Multitask Learners	2019	420
T5 Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer	2019	394
GPT Improving Language Understanding by Generative Pre-Training	2018	267
BART BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension	2019	253
Focus Focus Your Attention (with Adaptive IIR Filters)	2023	242
XLNet XLNet: Generalized Autoregressive Pretraining for Language Understanding	2019	144
ALBERT ALBERT: A Lite BERT for Self-supervised Learning of Language Representations	2019	143
DistilBERT DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter	2019	98
ELECTRA ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators	2020	89
Electric Pre-Training Transformers as Energy-Based Cloze Models	2020	81
Longformer Longformer: The Long-Document Transformer	2020	60
mT5 mT5: A massively multilingual pre-trained text-to-text transformer	2020	58
Transformer-XL Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	2019	53
XLM Cross-lingual Language Model Pretraining	2019	50
Performer Rethinking Attention with Performers	2020	49
mBART Multilingual Denoising Pre-training for Neural Machine Translation	2020	46
DeBERTa DeBERTa: Decoding-enhanced BERT with Disentangled Attention	2020	45
ERNIE ERNIE: Enhanced Representation through Knowledge Integration	2019	40
PaLM PaLM: Scaling Language Modeling with Pathways	2022	40
CodeBERT CodeBERT: A Pre-Trained Model for Programming and Natural Languages	2020	36
PEGASUS PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization	2019	29
Sparse Transformer Generating Long Sequences with Sparse Transformers	2019	27
ViLBERT ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks	2019	24
ETC ETC: Encoding Long and Structured Inputs in Transformers	2020	24
CodeT5 CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation	2021	21
GPT-Neo	2000	19
CTRL CTRL: A Conditional Transformer Language Model for Controllable Generation	2019	19
RAG Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks	2020	15
Switch Transformer Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity	2021	14
Reformer Reformer: The Efficient Transformer	2020	14
Universal Transformer Universal Transformers	2018	13
BigBird Big Bird: Transformers for Longer Sequences	2020	12
Linformer Linformer: Self-Attention with Linear Complexity	2020	12
Levenshtein Transformer Levenshtein Transformer	2019	10
Primer Primer: Searching for Efficient Transformers for Language Modeling	2021	9
ProphetNet ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training	2020	7
MATE MATE: Multi-view Attention for Table Transformer Efficiency	2021	6
PLATO-2 PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learning	2020	6
TNT Transformer in Transformer	2021	6
Bort Optimal Subarchitecture Extraction For BERT	2020	5
Charformer Charformer: Fast Character Transformers via Gradient-based Subword Tokenization	2021	5
MobileBERT MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices	2020	5
EGT Global Self-Attention as a Replacement for Graph Convolution	2021	5
E-Branchformer E-Branchformer: Branchformer with Enhanced merging for speech recognition	2022	4
Compressive Transformer Compressive Transformers for Long-Range Sequence Modelling	2019	3
Routing Transformer Efficient Content-Based Sparse Attention with Routing Transformers	2020	3
Adaptive Span Transformer Adaptive Attention Span in Transformers	2019	3
GANformer Generative Adversarial Transformers	2021	3
VideoBERT VideoBERT: A Joint Model for Video and Language Representation Learning	2019	3
DeeBERT DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference	2020	2
TAPEX TAPEX: Table Pre-training via Learning a Neural SQL Executor	2021	2
CuBERT Learning and Evaluating Contextual Embedding of Source Code	2019	2
Subformer Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers	2021	2
DynaBERT DynaBERT: Dynamic BERT with Adaptive Width and Depth	2020	2
TernaryBERT TernaryBERT: Distillation-aware Ultra-low Bit BERT	2020	2
Fastformer Fastformer: Additive Attention Can Be All You Need	2021	2
Nyströmformer Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention	2021	2
AutoTinyBERT AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models	2021	2
PermuteFormer PermuteFormer: Efficient Relative Position Encoding for Long Sequences	2021	1
NormFormer NormFormer: Improved Transformer Pretraining with Extra Normalization	2021	1
BP-Transformer BP-Transformer: Modelling Long-Range Context via Binary Partitioning	2019	1
IB-BERT MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices	2020	1
MacBERT Revisiting Pre-Trained Models for Chinese Natural Language Processing	2020	1
SMITH Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching	2020	1
ClipBERT Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling	2021	1
I-BERT I-BERT: Integer-only BERT Quantization	2021	1
SqueezeBERT SqueezeBERT: What can computer vision teach NLP about efficient neural networks?	2020	1
Sandwich Transformer Improving Transformer Models by Reordering their Sublayers	2019	1
DeLighT DeLighT: Deep and Light-weight Transformer	2020	1
PAR Transformer Pay Attention when Required	2020	1
ConvBERT ConvBERT: Improving BERT with Span-based Dynamic Convolution	2020	1
ESACL Enhanced Seq2Seq Autoencoder via Contrastive Learning for Abstractive Text Summarization	2021	1
MHMA Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos	2021	1
RealFormer RealFormer: Transformer Likes Residual Attention	2020	1
Sinkhorn Transformer Sparse Sinkhorn Attention	2020	1
SongNet SongNet: Rigid Formats Controlled Text Generation	2020	1
Funnel Transformer Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing	2020	1
T-D Generating Wikipedia by Summarizing Long Sequences	2018	1
SC-GPT Few-shot Natural Language Generation for Task-Oriented Dialog	2020	1
Chinese Pre-trained Unbalanced Transformer CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation	2021	1
BinaryBERT BinaryBERT: Pushing the Limit of BERT Quantization	2020	1
Adaptively Sparse Transformer Adaptively Sparse Transformers	2019	1
Feedback Transformer Addressing Some Limitations of Transformers with Feedback Memory	2020	1

Transformers Edit

Methods Add a Method

Transformers

Edit

Methods

Add a Method