no code implementations • CCL 2022 • Junjie Ye, Junjun Guo, Kaiwen Tan, Yan Xiang, Zhengtao Yu
“多模态神经机器翻译旨在利用视觉信息来提高文本翻译质量。传统多模态机器翻译将图像的全局语义信息融入到翻译模型, 而忽略了图像的细粒度信息对翻译质量的影响。对此, 该文提出一种基于图文细粒度对齐语义引导的多模态神经机器翻译方法, 该方法首先跨模态交互图文信息, 以提取图文细粒度对齐语义信息, 然后以图文细粒度对齐语义信息为枢纽, 采用门控机制将多模态细粒度信息对齐到文本信息上, 实现图文多模态特征融合。在多模态机器翻译基准数据集Multi30K 英语→德语、英语→法语以及英语→捷克语翻译任务上的实验结果表明, 论文提出方法的有效性, 并且优于大多数最先进的多模态机器翻译方法。”
no code implementations • COLING 2022 • Junjie Ye, Junjun Guo, Yan Xiang, Kaiwen Tan, Zhengtao Yu
This paper proposes a noise-robust multi-modal interactive fusion approach with cross-modal relation-aware mask mechanism for MNMT.
no code implementations • 22 Jan 2022 • Kaiwen Tan, Weixian Huang, Xiaofeng Liu, Jinlong Hu, Shoubin Dong
By integrating these heterogeneous but complementary data, many multi-modal methods are proposed to study the complex mechanisms of cancers, and most of them achieve comparable or better results from previous single-modal methods.