Named Entity and Relation Extraction with Multi-Modal Retrieval

3 Dec 2022  ·  Xinyu Wang, Jiong Cai, Yong Jiang, Pengjun Xie, Kewei Tu, Wei Lu ·

Multi-modal named entity recognition (NER) and relation extraction (RE) aim to leverage relevant image information to improve the performance of NER and RE. Most existing efforts largely focused on directly extracting potentially useful information from images (such as pixel-level features, identified objects, and associated captions). However, such extraction processes may not be knowledge aware, resulting in information that may not be highly relevant. In this paper, we propose a novel Multi-modal Retrieval based framework (MoRe). MoRe contains a text retrieval module and an image-based retrieval module, which retrieve related knowledge of the input text and image in the knowledge corpus respectively. Next, the retrieval results are sent to the textual and visual models respectively for predictions. Finally, a Mixture of Experts (MoE) module combines the predictions from the two models to make the final decision. Our experiments show that both our textual model and visual model can achieve state-of-the-art performance on four multi-modal NER datasets and one multi-modal RE dataset. With MoE, the model performance can be further improved and our analysis demonstrates the benefits of integrating both textual and visual cues for such tasks.

PDF Abstract

Datasets


  Add Datasets introduced or used in this paper
Task Dataset Model Metric Name Metric Value Global Rank Benchmark
Multi-modal Named Entity Recognition SNAP (MNER) MoRe-Text F1 90.09 # 4
Multi-modal Named Entity Recognition SNAP (MNER) MoRe-MoE F1 91.10 # 1
Multi-modal Named Entity Recognition SNAP (MNER) MoRe-Image F1 90.20 # 2
Multi-modal Named Entity Recognition SNAP (MNER) BERT-CRF F1 89.65 # 5
Multi-modal Named Entity Recognition Twitter-15 MoRe-MoE F1 79.21 # 2
Multi-modal Named Entity Recognition Twitter-15 MoRe-Text F1 77.91 # 4
Multi-modal Named Entity Recognition Twitter-15 BERT-CRF F1 77.04 # 5
Multi-modal Named Entity Recognition Twitter-15 MoRe-Image F1 78.13 # 3
Multi-modal Named Entity Recognition Twitter-17 MoRe-MoE F1 90.67 # 2
Multi-modal Named Entity Recognition Twitter-2017 MoRe-MoE F1 90.67 # 2
Multi-modal Named Entity Recognition WikiDiverse MoRe-Text F1 77.97 # 2
Multi-modal Named Entity Recognition WikiDiverse MoRe-Image F1 77.46 # 3
Multi-modal Named Entity Recognition WikiDiverse MoRe-MoE F1 79.33 # 1
Multi-modal Named Entity Recognition WikiDiverse BERT-CRF F1 76.58 # 5

Methods


No methods listed for this paper. Add relevant methods here