mT5: A massively multilingual pre-trained text-to-text transformer

The recent "Text-to-Text Transfer Transformer" (T5) leveraged a unified text-to-text format and scale to attain state-of-the-art results on a wide variety of English-language NLP tasks. In this paper, we introduce mT5, a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. We detail the design and modified training of mT5 and demonstrate its state-of-the-art performance on many multilingual benchmarks. We also describe a simple technique to prevent "accidental translation" in the zero-shot setting, where a generative model chooses to (partially) translate its prediction into the wrong language. All of the code and model checkpoints used in this work are publicly available.

PDF Abstract NAACL 2021 PDF NAACL 2021 Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Question Answering DaNetQA MT5 Large Accuracy 0.657 # 10
Natural Language Inference LiDiRus MT5 Large MCC 0.061 # 16
Reading Comprehension MuSeRC MT5 Large Average F1 0.844 # 2
EM 0.543 # 4
Common Sense Reasoning PARus MT5 Large Accuracy 0.504 # 14
Natural Language Inference RCB MT5 Large Average F1 0.366 # 11
Accuracy 0.454 # 15
Common Sense Reasoning RuCoS MT5 Large Average F1 0.57 # 10
EM 0.562 # 10
Common Sense Reasoning RWSD MT5 Large Accuracy 0.669 # 8
Natural Language Inference TERRa MT5 Large Accuracy 0.561 # 16
Zero-Shot Cross-Lingual Transfer XTREME mT5 Sentence-pair Classification 89.8 # 4
Structured Prediction NA # 19
Question Answering 73.6 # 5
Sentence Retrieval NA # 18
Avg 40.9 # 18