CC100

Introduced by Conneau et al. in Unsupervised Cross-lingual Representation Learning at Scale

This corpus comprises of monolingual data for 100+ languages and also includes data for romanized languages. This was constructed using the urls and paragraph indices provided by the CC-Net repository by processing January-December 2018 Commoncrawl snapshots. Each file comprises of documents separated by double-newlines and paragraphs within the same document separated by a newline. The data is generated using the open source CC-Net repository.

Source: Unsupervised Cross-lingual Representation Learning at Scale

Homepage