Browse > Natural Language Processing > Machine Translation > Multimodal Machine Translation

Multimodal Machine Translation

7 papers with code ยท Natural Language Processing
Subtask of Machine Translation

Multimodal machine translation is the task of doing machine translation with multiple data sources - for example, translating "a bird is flying over water" + an image of a bird over water to German text.

State-of-the-art leaderboards

No evaluation results yet. Help compare methods by submit evaluation metrics.

Latest papers without code

On Leveraging the Visual Modality for Neural Machine Translation

7 Oct 2019

Leveraging the visual modality effectively for Neural Machine Translation (NMT) remains an open problem in computational linguistics.

MULTIMODAL MACHINE TRANSLATION

Probing Representations Learned by Multimodal Recurrent and Transformer Models

29 Aug 2019

In this paper, we present a meta-study assessing the representational quality of models where the training signal is obtained from different modalities, in particular, language modeling, image features prediction, and both textual and multimodal machine translation.

IMAGE RETRIEVAL LANGUAGE MODELLING MULTIMODAL MACHINE TRANSLATION SEMANTIC TEXTUAL SIMILARITY

Hindi Visual Genome: A Dataset for Multimodal English-to-Hindi Machine Translation

21 Jul 2019

We present ``Hindi Visual Genome'', a multimodal dataset consisting of text and images suitable for English-Hindi multimodal machine translation task and multimodal research.

MULTIMODAL MACHINE TRANSLATION

Distilling Translations with Visual Awareness

ACL 2019

Previous work on multimodal machine translation has shown that visual information is only needed in very specific cases, for example in the presence of ambiguous words where the textual context is not sufficient.

MULTIMODAL MACHINE TRANSLATION

Distilling Translations with Visual Awareness

ACL 2019

Previous work on multimodal machine translation has shown that visual information is only needed in very specific cases, for example in the presence of ambiguous words where the textual context is not sufficient.

MULTIMODAL MACHINE TRANSLATION

Probing the Need for Visual Context in Multimodal Machine Translation

NAACL 2019

Current work on multimodal machine translation (MMT) has suggested that the visual modality is either unnecessary or only marginally beneficial.

MULTIMODAL MACHINE TRANSLATION

Multimodal Machine Translation with Embedding Prediction

NAACL 2019

Multimodal machine translation is an attractive application of neural machine translation (NMT).

MULTIMODAL MACHINE TRANSLATION WORD EMBEDDINGS

Grounded Word Sense Translation

NAACL 2019

Recent work on visually grounded language learning has focused on broader applications of grounded representations, such as visual question answering and multimodal machine translation.

MULTIMODAL MACHINE TRANSLATION QUESTION ANSWERING VISUAL QUESTION ANSWERING

Debiasing Word Embeddings Improves Multimodal Machine Translation

24 May 2019

In this study, we examine various kinds of word embeddings and introduce two debiasing techniques for three multimodal NMT models and two language pairs -- English-German translation and English-French translation.

MULTIMODAL MACHINE TRANSLATION WORD EMBEDDINGS

Probing the Need for Visual Context in Multimodal Machine Translation

NAACL 2019

Current work on multimodal machine translation (MMT) has suggested that the visual modality is either unnecessary or only marginally beneficial.

MULTIMODAL MACHINE TRANSLATION