In state-of-the-art Neural Machine Translation (NMT), an attention mechanism
is used during decoding to enhance the translation. At every step, the decoder
uses this mechanism to focus on different parts of the source sentence to
gather the most useful information before outputting its target word...
the effectiveness of the attention mechanism has also been explored for
multimodal tasks, where it becomes possible to focus both on sentence parts and
image regions that they describe. In this paper, we compare several attention
mechanism on the multimodal translation task (English, image to German) and
evaluate the ability of the model to make use of images to improve translation. We surpass state-of-the-art scores on the Multi30k data set, we nevertheless
identify and report different misbehavior of the machine while translating.