Language Modeling is the task of predicting the next word or character in a document. This technique can be used to train language models that can further be applied to a wide range of natural language tasks like text generation, text classification, and question answering.

The common types of language modeling techniques involve:

• N-gram Language Models
• Neural Langauge Models

A model's language modeling capability is measured using cross-entropy and perplexity. Some datasets to evaluate language modeling are WikiText-103, One Billion Word, Text8, C4, among others.

One of the most recent popular benchmarks to evaluate language modeling capabilities is called SuperGLUE.

Some popular and notable state-of-the-art language models, include:

• GPT-3
• Megatron-LM
• BERT

Check below for all state-of-the-art models.

Here are some additional readings to go deeper on the task:

• Language Modeling - Lena Voita

( Image credit: Exploring the Limits of Language Modeling )