XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
In this paper, we introduce ELECTRA-style tasks to cross-lingual language model pre-training. Specifically, we present two pre-training tasks, namely multilingual replaced token detection, and translation replaced token detection. Besides, we pretrain the model, named as XLM-E, on both multilingual and parallel corpora. Our model outperforms the baseline models on various cross-lingual understanding tasks with much less computation cost. Moreover, analysis shows that XLM-E tends to obtain better cross-lingual transferability.
PDF Abstract ACL 2022 PDF ACL 2022 AbstractCode
Spaces
XNLI
XQuAD
PAWS-X
TyDiQA
MLQA
XTREME
| Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Benchmark |
|---|---|---|---|---|---|---|
| Zero-Shot Cross-Lingual Transfer | XTREME | Turing ULR v6 | Sentence-pair Classification | 91.0 | # 1 | |
| Structured Prediction | 83.8 | # 2 | ||||
| Question Answering | 77.1 | # 1 | ||||
| Sentence Retrieval | 94.4 | # 1 | ||||
| Avg | 85.5 | # 1 | ||||
| Zero-Shot Cross-Lingual Transfer | XTREME | Turing ULR v5 | Sentence-pair Classification | 90.3 | # 4 | |
| Structured Prediction | 81.7 | # 5 | ||||
| Question Answering | 76.3 | # 2 | ||||
| Sentence Retrieval | 93.7 | # 5 | ||||
| Avg | 84.5 | # 4 |
Methods
No methods listed for this paper. Add
relevant methods here
