Browse > Natural Language Processing > Language Modelling

Language Modelling

424 papers with code · Natural Language Processing

Language modeling is the task of predicting the next word or character in a document.

* indicates models using dynamic evaluation; where, at test time, models may adapt to seen tokens in order to improve performance on following tokens. (Mikolov et al., (2010), Kraus et al., (2017))

Leaderboards

Latest papers without code

XD: Cross-lingual Knowledge Distillation for Polyglot Sentence Embeddings

ICLR 2020

Current state-of-the-art results in multilingual natural language inference (NLI) are based on tuning XLM (a pre-trained polyglot language model) separately for each language involved, resulting in multiple models.

LANGUAGE MODELLING NATURAL LANGUAGE INFERENCE SENTENCE EMBEDDINGS

Improving the Gating Mechanism of Recurrent Neural Networks

ICLR 2020

In this work, we revisit the gating mechanisms widely used in various recurrent and feedforward networks such as LSTMs, GRUs, or highway networks.

LANGUAGE MODELLING SEQUENTIAL IMAGE CLASSIFICATION

Extreme Language Model Compression with Optimal Subwords and Shared Projections

ICLR 2020

In particular, the input word embedding matrix accounts for a significant proportion of the model's memory footprint, due to the large input vocabulary and embedding dimensions.

LANGUAGE MODELLING MODEL COMPRESSION WORD EMBEDDINGS

Regularizing activations in neural networks via distribution matching with the Wassertein metric

ICLR 2020

We propose the projected error function regularization loss (PER) that encourages activations to follow the standard normal distribution.

IMAGE CLASSIFICATION LANGUAGE MODELLING

Masked Translation Model

ICLR 2020

We introduce the masked translation model (MTM) which combines encoding and decoding of sequences within the same model component.

LANGUAGE MODELLING

Resolving Lexical Ambiguity in English–Japanese Neural Machine Translation

ICLR 2020

Lexical ambiguity, i. e., the presence of two or more meanings for a single word, is an inherent and challenging problem for machine translation systems.

LANGUAGE MODELLING MACHINE TRANSLATION WORD EMBEDDINGS

Residual Energy-Based Models for Text Generation

ICLR 2020

In this work, we investigate un-normalized energy-based models (EBMs) which operate not at the token but at the sequence level.

LANGUAGE MODELLING MACHINE TRANSLATION TEXT GENERATION

Permutation Equivariant Models for Compositional Generalization in Language

ICLR 2020

Humans understand novel sentences by composing meanings and roles of core language components.

LANGUAGE MODELLING

StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding

ICLR 2020

Recently, the pre-trained language model, BERT (and its robustly optimized version RoBERTa), has attracted a lot of attention in natural language understanding (NLU), and achieved state-of-the-art accuracy in various NLU tasks, such as sentiment classification, natural language inference, semantic textual similarity and question answering.

LANGUAGE MODELLING NATURAL LANGUAGE INFERENCE QUESTION ANSWERING SEMANTIC TEXTUAL SIMILARITY SENTIMENT ANALYSIS

Lossless Data Compression with Transformer

ICLR 2020

It is closely related to the problem of online learning of language models.

LANGUAGE MODELLING MACHINE TRANSLATION