Autoencoding Transformers

Edit

Natural Language Processing • Transformers • 28 methods

Methods

Add a Method

Method	Year	Papers
BERT BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding	2018	5063
RoBERTa RoBERTa: A Robustly Optimized BERT Pretraining Approach	2019	747
T5 Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer	2019	553
ALBERT ALBERT: A Lite BERT for Self-supervised Learning of Language Representations	2019	154
DistilBERT DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter	2019	126
Electric Pre-Training Transformers as Energy-Based Cloze Models	2020	118
ELECTRA ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators	2020	107
mT5 mT5: A massively multilingual pre-trained text-to-text transformer	2020	81
Longformer Longformer: The Long-Document Transformer	2020	78
DeBERTa DeBERTa: Decoding-enhanced BERT with Disentangled Attention	2020	65
mBART Multilingual Denoising Pre-training for Neural Machine Translation	2020	59
XLM Cross-lingual Language Model Pretraining	2019	55
CodeT5 CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation	2021	18
Switch Transformer Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity	2021	16
MobileBERT MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices	2020	5
Bort Optimal Subarchitecture Extraction For BERT	2020	5
DeeBERT DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference	2020	3
ConvBERT ConvBERT: Improving BERT with Span-based Dynamic Convolution	2020	3
I-BERT I-BERT: Integer-only BERT Quantization	2021	2
CuBERT Learning and Evaluating Contextual Embedding of Source Code	2019	2
DynaBERT DynaBERT: Dynamic BERT with Adaptive Width and Depth	2020	2
TernaryBERT TernaryBERT: Distillation-aware Ultra-low Bit BERT	2020	2
AutoTinyBERT AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models	2021	2
IB-BERT MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices	2020	1
SqueezeBERT SqueezeBERT: What can computer vision teach NLP about efficient neural networks?	2020	1
SMITH Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching	2020	1
MacBERT Revisiting Pre-Trained Models for Chinese Natural Language Processing	2020	1
BinaryBERT BinaryBERT: Pushing the Limit of BERT Quantization	2020	1

Autoencoding Transformers Edit

Methods Add a Method

Autoencoding Transformers

Edit

Methods

Add a Method