Language Models

Edit

Natural Language Processing • 141 methods

Language Models are models for predicting the next word or character in a document. Below you can find a continuously updating list of language models.

Subcategories

1 Transformers

Methods

Add a Method

Method	Year	Papers
Transformer Attention Is All You Need	2017	12120
Focus Focus Your Attention (with Adaptive IIR Filters)	2023	10724
Diffusion Denoising Diffusion Probabilistic Models	2020	10071
BERT BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding	2018	5982
GPT-4 GPT-4 Technical Report	2023	2439
GPT-3 Language Models are Few-Shot Learners	2020	1760
GPT Improving Language Understanding by Generative Pre-Training	2018	992
BART BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension	2019	973
RoBERTa RoBERTa: A Robustly Optimized BERT Pretraining Approach	2019	845
LLaMA LLaMA: Open and Efficient Foundation Language Models	2023	709
GPT-2 Language Models are Unsupervised Multitask Learners	2019	676
RAG Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks	2020	646
T5 Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer	2019	642
OPT OPT: Open Pre-trained Transformer Language Models	2022	247
CAM Is Object Localization for Free? - Weakly-Supervised Learning With Convolutional Neural Networks	2015	240
ELMo Deep contextualized word representations	2018	230
mBERT BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding	2018	192
XLM-R Unsupervised Cross-lingual Representation Learning at Scale	2019	169
ALBERT ALBERT: A Lite BERT for Self-supervised Learning of Language Representations	2019	165
XLNet XLNet: Generalized Autoregressive Pretraining for Language Understanding	2019	162
DistilBERT DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter	2019	146
Electric Pre-Training Transformers as Energy-Based Cloze Models	2020	142
PaLM PaLM: Scaling Language Modeling with Pathways	2022	137
ELECTRA ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators	2020	111
BLOOM BLOOM: A 176B-Parameter Open-Access Multilingual Language Model	2022	103
mT5 mT5: A massively multilingual pre-trained text-to-text transformer	2020	95
Flan-T5 Scaling Instruction-Finetuned Language Models	2022	95
Performer Rethinking Attention with Performers	2020	90
Longformer Longformer: The Long-Document Transformer	2020	83
DeBERTa DeBERTa: Decoding-enhanced BERT with Disentangled Attention	2020	79
mBART Multilingual Denoising Pre-training for Neural Machine Translation	2020	64
Transformer-XL Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	2019	63
CodeBERT CodeBERT: A Pre-Trained Model for Programming and Natural Languages	2020	59
XLM Cross-lingual Language Model Pretraining	2019	56
ERNIE ERNIE: Enhanced Representation through Knowledge Integration	2019	54
PEGASUS PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization	2019	50
Pythia Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling	2023	45
Sparse Transformer Generating Long Sequences with Sparse Transformers	2019	44
GLM GLM-130B: An Open Bilingual Pre-trained Model	2022	40
ULMFiT Universal Language Model Fine-tuning for Text Classification	2018	39
GPT-Neo	2000	36
ETC ETC: Encoding Long and Structured Inputs in Transformers	2020	33
ViLBERT ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks	2019	30
CTRL CTRL: A Conditional Transformer Language Model for Controllable Generation	2019	25
BLOOMZ Crosslingual Generalization through Multitask Finetuning	2022	25
CodeGen CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis	2022	24
Synthesizer Synthesizer: Rethinking Self-Attention in Transformer Models	2020	23
CodeT5 CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation	2021	23
Chinchilla Training Compute-Optimal Large Language Models	2022	20
Universal Transformer Universal Transformers	2018	18
Switch Transformer Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity	2021	17
Reformer Reformer: The Efficient Transformer	2020	17
Linformer Linformer: Self-Attention with Linear Complexity	2020	16
BigBird Big Bird: Transformers for Longer Sequences	2020	16
TNT Transformer in Transformer	2021	12
E-Branchformer E-Branchformer: Branchformer with Enhanced merging for speech recognition	2022	12
Levenshtein Transformer Levenshtein Transformer	2019	12
GPT-NeoX GPT-NeoX-20B: An Open-Source Autoregressive Language Model	2022	11
ProphetNet ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training	2020	11
Primer Primer: Searching for Efficient Transformers for Language Modeling	2021	11
Galactica Galactica: A Large Language Model for Science	2022	10
MATE MATE: Multi-view Attention for Table Transformer Efficiency	2021	10
MobileBERT MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices	2020	9
GEE Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes	2021	8
UL2 UL2: Unifying Language Learning Paradigms	2022	8
CharacterBERT CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters	2020	8
Cross-View Training Semi-Supervised Sequence Modeling with Cross-View Training	2018	8
mT0 Crosslingual Generalization through Multitask Finetuning	2022	7
EGT Global Self-Attention as a Replacement for Graph Convolution	2021	7
PLATO-2 PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learning	2020	6
Neural Probabilistic Language Model	2003	5
Bort Optimal Subarchitecture Extraction For BERT	2020	5
Charformer Charformer: Fast Character Transformers via Gradient-based Subword Tokenization	2021	5
CANINE CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation	2021	5
Gated Convolution Network Language Modeling with Gated Convolutional Networks	2016	3
I-BERT I-BERT: Integer-only BERT Quantization	2021	3
Cross-encoder Reranking ReRankMatch: Semi-Supervised Learning with Semantics-Oriented Similarity Representation	2021	3
OPT-IML OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization	2022	3
Fastformer Fastformer: Additive Attention Can Be All You Need	2021	3
Compressive Transformer Compressive Transformers for Long-Range Sequence Modelling	2019	3
Routing Transformer Efficient Content-Based Sparse Attention with Routing Transformers	2020	3
DeeBERT DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference	2020	3
Subformer Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers	2021	3
Adaptive Span Transformer Adaptive Attention Span in Transformers	2019	3
GANformer Generative Adversarial Transformers	2021	3
VideoBERT VideoBERT: A Joint Model for Video and Language Representation Learning	2019	3
TAPEX TAPEX: Table Pre-training via Learning a Neural SQL Executor	2021	3
ConvBERT ConvBERT: Improving BERT with Span-based Dynamic Convolution	2020	3
Feedback Transformer Addressing Some Limitations of Transformers with Feedback Memory	2020	2
PMLM Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order	2020	2
CuBERT Learning and Evaluating Contextual Embedding of Source Code	2019	2
DynaBERT DynaBERT: Dynamic BERT with Adaptive Width and Depth	2020	2
TernaryBERT TernaryBERT: Distillation-aware Ultra-low Bit BERT	2020	2
Parallel Layers PaLM: Scaling Language Modeling with Pathways	2022	2
ESACL Enhanced Seq2Seq Autoencoder via Contrastive Learning for Abstractive Text Summarization	2021	2
MacBERT Revisiting Pre-Trained Models for Chinese Natural Language Processing	2020	2
SHA-RNN Single Headed Attention RNN: Stop Thinking With Your Head	2019	2
Sandwich Transformer Improving Transformer Models by Reordering their Sublayers	2019	2
Nyströmformer Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention	2021	2
AutoTinyBERT AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models	2021	2
PermuteFormer PermuteFormer: Efficient Relative Position Encoding for Long Sequences	2021	1
NormFormer NormFormer: Improved Transformer Pretraining with Extra Normalization	2021	1
BP-Transformer BP-Transformer: Modelling Long-Range Context via Binary Partitioning	2019	1
IB-BERT MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices	2020	1
CPM-2 CPM-2: Large-scale Cost-effective Pre-trained Language Models	2021	1
MixLoRA MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts	2024	1
mBARTHez BARThez: a Skilled Pretrained French Sequence-to-Sequence Model	2020	1
ooJpiued Dynamic Network Model from Partial Observations	2018	1
PanGu-$α$ PanGu-$α$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation	2021	1
Step-DPO	2000	1
KE-MLM Knowledge Enhanced Masked Language Model for Stance Detection	2021	1
SMITH Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching	2020	1
ClipBERT Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling	2021	1
SqueezeBERT SqueezeBERT: What can computer vision teach NLP about efficient neural networks?	2020	1
ERNIE-GEN ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation	2020	1
DeLighT DeLighT: Deep and Light-weight Transformer	2020	1
PAR Transformer Pay Attention when Required	2020	1
PolyNorm Polynomial Composition Activations: Unleashing the Dynamics of Large Language Models	2024	1
MHMA Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos	2021	1
RealFormer RealFormer: Transformer Likes Residual Attention	2020	1
Sinkhorn Transformer Sparse Sinkhorn Attention	2020	1
SongNet SongNet: Rigid Formats Controlled Text Generation	2020	1
Funnel Transformer Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing	2020	1
T-D Generating Wikipedia by Summarizing Long Sequences	2018	1
SC-GPT Few-shot Natural Language Generation for Task-Oriented Dialog	2020	1
Chinese Pre-trained Unbalanced Transformer CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation	2021	1
BinaryBERT BinaryBERT: Pushing the Limit of BERT Quantization	2020	1
Adaptively Sparse Transformer Adaptively Sparse Transformers	2019	1

Language Models Edit

Methods Add a Method

Language Models

Edit

Methods

Add a Method