Cross-lingual Language Model Pretraining

NeurIPS 2019  ·  Guillaume Lample, Alexis Conneau ·

Recent studies have demonstrated the efficiency of generative pretraining for English natural language understanding. In this work, we extend this approach to multiple languages and show the effectiveness of cross-lingual pretraining. We propose two methods to learn cross-lingual language models (XLMs): one unsupervised that only relies on monolingual data, and one supervised that leverages parallel data with a new cross-lingual language model objective. We obtain state-of-the-art results on cross-lingual classification, unsupervised and supervised machine translation. On XNLI, our approach pushes the state of the art by an absolute gain of 4.9% accuracy. On unsupervised machine translation, we obtain 34.3 BLEU on WMT'16 German-English, improving the previous state of the art by more than 9 BLEU. On supervised machine translation, we obtain a new state of the art of 38.5 BLEU on WMT'16 Romanian-English, outperforming the previous best approach by more than 4 BLEU. Our code and pretrained models will be made publicly available.

PDF Abstract NeurIPS 2019 PDF NeurIPS 2019 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Unsupervised Machine Translation WMT2014 English-French MLM pretraining for encoder and decoder BLEU 33.4 # 4
Unsupervised Machine Translation WMT2014 French-English MLM pretraining for encoder and decoder BLEU 33.3 # 4
Unsupervised Machine Translation WMT2016 English-German MLM pretraining for encoder and decoder BLEU 26.4 # 4
Unsupervised Machine Translation WMT2016 English-Romanian MLM pretraining for encoder and decoder BLEU 33.3 # 2
Unsupervised Machine Translation WMT2016 English--Romanian MLM pretraining for encoder and decoder BLEU 33.3 # 2
Unsupervised Machine Translation WMT2016 German-English MLM pretraining for encoder and decoder BLEU 34.3 # 4
Unsupervised Machine Translation WMT2016 Romanian-English MLM pretraining for encoder and decoder BLEU 31.8 # 3
Machine Translation WMT2016 Romanian-English MLM pretraining BLEU score 35.3 # 4
Natural Language Inference XNLI French XLM (MLM+TLM) Accuracy 80.2 # 4

Methods