Endowing Language Models with Multimodal Knowledge Graph Representations

1 code implementation • 27 Jun 2022 • Ningyuan Huang, Yash R. Deshpande, Yibo Liu, Houda Alberts, Kyunghyun Cho, Clara Vania, Iacer Calixto

We use the recently released VisualSem KG as our external knowledge repository, which covers a subset of Wikipedia and WordNet entities, and compare a mix of tuple-based and graph-based algorithms to learn entity and relation representations that are grounded on the KG multimodal information.

Multilingual Named Entity Recognition named-entity-recognition +2

VisualSem: A High-quality Knowledge Graph for Vision and Language

1 code implementation • EMNLP (MRL) 2021 • Houda Alberts, Teresa Huang, Yash Deshpande, Yibo Liu, Kyunghyun Cho, Clara Vania, Iacer Calixto

We also release a neural multi-modal retrieval model that can use images or sentences as inputs and retrieves entities in the KG.

Data Augmentation Natural Language Understanding +2

ImagiFilter: A resource to enable the semi-automatic mining of images at scale

1 code implementation • 20 Aug 2020 • Houda Alberts, Iacer Calixto

In this paper, we describe and publicly release an image dataset along with pretrained models designed to (semi-)automatically filter out undesirable images from very large image collections, possibly obtained from the web.