Cross-lingual Language Model Pretraining

22 Jan 2019Guillaume LampleAlexis Conneau

Recent studies have demonstrated the efficiency of generative pretraining for English natural language understanding. In this work, we extend this approach to multiple languages and show the effectiveness of cross-lingual pretraining... (read more)

PDF Abstract
Task Dataset Model Metric name Metric value Global rank Compare
Unsupervised Machine Translation WMT2014 English-French MLM pretraining for encoder and decoder BLEU 33.4 # 3
Unsupervised Machine Translation WMT2014 French-English MLM pretraining for encoder and decoder BLEU 33.3 # 3
Unsupervised Machine Translation WMT2016 English-German MLM pretraining for encoder and decoder BLEU 26.4 # 3
Unsupervised Machine Translation WMT2016 English-Romanian MLM pretraining for encoder and decoder BLEU 33.3 # 1
Unsupervised Machine Translation WMT2016 English--Romanian MLM pretraining for encoder and decoder BLEU 33.3 # 2
Unsupervised Machine Translation WMT2016 German-English MLM pretraining for encoder and decoder BLEU 34.3 # 3
Unsupervised Machine Translation WMT2016 Romanian-English MLM pretraining for encoder and decoder BLEU 31.8 # 2
Machine Translation WMT2016 Romanian-English MLM pretraining BLEU score 35.3 # 1