Introduced by Lample et al. in Cross-lingual Language Model Pretraining

XLM is a Transformer based architecture that is pre-trained using one of three language modelling objectives:

  1. Causal Language Modeling - models the probability of a word given the previous words in a sentence.
  2. Masked Language Modeling - the masked language modeling objective of BERT.
  3. Translation Language Modeling - a (new) translation language modeling objective for improving cross-lingual pre-training.

The authors find that both the CLM and MLM approaches provide strong cross-lingual features that can be used for pretraining models.

Source: Cross-lingual Language Model Pretraining


Paper Code Results Date Stars


Task Papers Share
Language Modelling 17 8.90%
Translation 15 7.85%
Sentence 11 5.76%
Machine Translation 10 5.24%
XLM-R 8 4.19%
Cross-Lingual Transfer 8 4.19%
Question Answering 8 4.19%
Retrieval 6 3.14%
NER 6 3.14%