Multilingual Multimodal Learning with Machine Translated Text

24 Oct 2022  ·  Chen Qiu, Dan Oneata, Emanuele Bugliarello, Stella Frank, Desmond Elliott ·

Most vision-and-language pretraining research focuses on English tasks. However, the creation of multilingual multimodal evaluation datasets (e.g. Multi30K, xGQA, XVNLI, and MaRVL) poses a new challenge in finding high-quality training data that is both multilingual and multimodal. In this paper, we investigate whether machine translating English multimodal data can be an effective proxy for the lack of readily available multilingual data. We call this framework TD-MML: Translated Data for Multilingual Multimodal Learning, and it can be applied to any multimodal dataset and model. We apply it to both pretraining and fine-tuning data with a state-of-the-art model. In order to prevent models from learning from low-quality translated text, we propose two metrics for automatically removing such translations from the resulting datasets. In experiments on five tasks across 20 languages in the IGLUE benchmark, we show that translated data can provide a useful signal for multilingual multimodal learning, both at pretraining and fine-tuning.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Zero-Shot Cross-Lingual Visual Reasoning MaRVL TD-MML (finetuned on English-only data) Accuracy (%) 59.67 # 5
Zero-Shot Cross-Lingual Visual Reasoning MaRVL TD-MML (finetuned on machine translated data) Accuracy (%) 61.15 # 4
Zero-Shot Cross-Lingual Text-to-Image Retrieval WIT (IGLUE) TD-MML Recall@1 (%) 9.76 # 1
Zero-Shot Cross-Lingual Image-to-Text Retrieval WIT (IGLUE) TD-MML Recall@1 (%) 10.40 # 2
Zero-Shot Cross-Lingual Image-to-Text Retrieval xFlickr&CO TD-MML Recall@1 (%) 26.35 # 4
Zero-Shot Cross-Lingual Text-to-Image Retrieval xFlickr&CO TD-MML Recall@1 (%) 21.3 # 4
Zero-Shot Cross-Lingual Visual Question Answering xGQA TD-MML (finetuned on machine translated data) Accuracy (%) 45.21 # 3
Zero-Shot Cross-Lingual Visual Question Answering xGQA TD-MML (finetuned on English-only data) Accuracy (%) 35.95 # 5
Zero-Shot Cross-Lingual Visual Natural Language Inference XVNLI TD-MML Accuracy (%) 64.84 # 5

Methods


No methods listed for this paper. Add relevant methods here