MultiFiT: Efficient Multi-lingual Language Model Fine-tuning

IJCNLP 2019  ·  Julian Martin Eisenschlos, Sebastian Ruder, Piotr Czapla, Marcin Kardas, Sylvain Gugger, Jeremy Howard ·

Pretrained language models are promising particularly for low-resource languages as they only require unlabelled data. However, training existing models requires huge amounts of compute, while pretrained cross-lingual models often underperform on low-resource languages. We propose Multi-lingual language model Fine-Tuning (MultiFiT) to enable practitioners to train and fine-tune language models efficiently in their own language. In addition, we propose a zero-shot method using an existing pretrained cross-lingual model. We evaluate our methods on two widely used cross-lingual classification datasets where they outperform models pretrained on orders of magnitude more data and compute. We release all models and code.

PDF Abstract IJCNLP 2019 PDF IJCNLP 2019 Abstract

Datasets

MLDoc
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Zero-shot Cross-Lingual Document Classification Cross-Lingual Sentiment (CLS)- English to French - Books MultiFiT, pseudo Accuracy 87.84 # 1
Compare
Zero-shot Cross-Lingual Document Classification Cross-Lingual Sentiment (CLS)- English to French - DVD MultiFiT, pseudo Accuracy 83.5 # 1
Compare
Zero-shot Cross-Lingual Document Classification Cross-Lingual Sentiment (CLS)- English to French - Music MultiFiT, pseudo Accuracy 85.6 # 1
Compare
Zero-shot Cross-Lingual Document Classification Cross-Lingual Sentiment (CLS)- English to German - Books MultiFiT, pseudo Accuracy 89.6 # 1
Compare
Zero-shot Cross-Lingual Document Classification Cross-Lingual Sentiment (CLS)- English to German - DVD MultiFiT, pseudo Accuracy 81.8 # 1
Compare
Zero-shot Cross-Lingual Document Classification Cross-Lingual Sentiment (CLS)- English to German - Music MultiFiT, pseudo Accuracy 84.4 # 1
Compare
Zero-shot Cross-Lingual Document Classification Cross-Lingual Sentiment (CLS)- English to Japanese - Books MultiFiT, pseudo Accuracy 80.45 # 1
Compare
Zero-shot Cross-Lingual Document Classification Cross-Lingual Sentiment (CLS)- English to Japanese - DVD MultiFiT, pseudo Accuracy 77.65 # 1
Compare
Zero-shot Cross-Lingual Document Classification Cross-Lingual Sentiment (CLS)- English to Japanese - Music MultiFiT, pseudo Accuracy 81.5 # 1
Compare
Cross-Lingual Document Classification MLDoc Zero-Shot English-to-Chinese MultiFiT, pseudo Accuracy 82.48 # 2
Compare
Cross-Lingual Document Classification MLDoc Zero-Shot English-to-French MultiFiT, pseudo Accuracy 89.42 # 2
Compare
Cross-Lingual Document Classification MLDoc Zero-Shot English-to-German MultiFiT, pseudo Accuracy 91.62% # 2
Compare
Cross-Lingual Document Classification MLDoc Zero-Shot English-to-Italian MultiFiT, pseudo Accuracy 76.02 # 1
Compare
Cross-Lingual Document Classification MLDoc Zero-Shot English-to-Japanese MultiFiT, pseudo Accuracy 69.57 # 1
Compare
Cross-Lingual Document Classification MLDoc Zero-Shot English-to-Russian MultiFiT, pseudo Accuracy 67.83 # 2
Compare
Cross-Lingual Document Classification MLDoc Zero-Shot English-to-Spanish MultiFiT, pseudo Accuracy 79.1 # 2
Compare

Methods
Add Remove

No methods listed for this paper. Add relevant methods here
Contact us on: hello@paperswithcode.com . Papers With Code is a free resource with all data licensed under CC-BY-SA.
Terms Data policy Cookies policy