Unsupervised Cross-lingual Representation Learning at Scale

5 Nov 2019Alexis ConneauKartikay KhandelwalNaman GoyalVishrav ChaudharyGuillaume WenzekFrancisco GuzmánEdouard GraveMyle OttLuke ZettlemoyerVeselin Stoyanov

This paper shows that pretraining multilingual language models at scale leads to significant performance gains for a wide range of cross-lingual transfer tasks. We train a Transformer-based masked language model on one hundred languages, using more than two terabytes of filtered CommonCrawl data... (read more)

PDF Abstract

Results from the Paper


  Submit results from this paper to get state-of-the-art GitHub badges and help the community compare results to other papers.