Search Results for author:
Found 29 papers, 9 papers with code
Multi3Generation: Multitask, Multilingual, Multimodal Language Generation
This paper presents the Multitask, Multilingual, Multimodal Language Generation COST Action – Multi3Generation (CA18231), an interdisciplinary network of research groups working on different aspects of language generation.
Fixing confirmation bias in feature attribution methods via semantic match
Feature attribution methods have become a staple method to disentangle the complex behavior of black box models.
Soft-prompt tuning to predict lung cancer using primary care free-text Dutch medical notes
We investigate different natural language processing (NLP) approaches based on contextualised word representations for the problem of early prediction of lung cancer using free-text patient medical notes of Dutch primary care physicians.
Detecting Euphemisms with Literal Descriptions and Visual Imagery
In the second stage, we integrate visual supervision into our system using visual imageries, two sets of images generated by a text-to-image model by taking terms and descriptions as input.
Endowing Language Models with Multimodal Knowledge Graph Representations
We use the recently released VisualSem KG as our external knowledge repository, which covers a subset of Wikipedia and WordNet entities, and compare a mix of tuple-based and graph-based algorithms to learn entity and relation representations that are grounded on the KG multimodal information.
Multilingual Named Entity Recognition
named-entity-recognition
+2
VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena
We propose VALSE (Vision And Language Structured Evaluation), a novel benchmark designed for testing general-purpose pretrained vision and language (V&L) models for their visio-linguistic grounding capabilities on specific linguistic phenomena.
Ranked #1 on
image-sentence alignment
on VALSE
Zero-shot hashtag segmentation for multilingual sentiment analysis
Hashtag segmentation, also known as hashtag decomposition, is a common step in preprocessing pipelines for social media datasets.
Wikipedia Entities as Rendezvous across Languages: Grounding Multilingual Language Models by Predicting Wikipedia Hyperlinks
Further adding extra languages lead to improvements in most tasks up to a certain point, but overall we found it non-trivial to scale improvements in model transferability by training on ever increasing amounts of Wikipedia languages.
Seeing past words: Testing the cross-modal capabilities of pretrained V&L models on counting tasks
We investigate the reasoning ability of pretrained vision and language (V&L) models in two tasks that require multimodal integration: (1) discriminating a correct image-sentence pair from an incorrect one, and (2) counting entities in an image.
A Study on the Autoregressive and non-Autoregressive Multi-label Learning
These tasks are hard because the label space is usually (i) very large, e. g. thousands or millions of labels, (ii) very sparse, i. e. very few labels apply to each input document, and (iii) highly correlated, meaning that the existence of one label changes the likelihood of predicting all other labels.
Are scene graphs good enough to improve Image Captioning?
Overall, we find no significant difference between models that use scene graph features and models that only use object detection features across different captioning metrics, which suggests that existing scene graph generation models are still too noisy to be useful in image captioning.
ImagiFilter: A resource to enable the semi-automatic mining of images at scale
In this paper, we describe and publicly release an image dataset along with pretrained models designed to (semi-)automatically filter out undesirable images from very large image collections, possibly obtained from the web.
VisualSem: A High-quality Knowledge Graph for Vision and Language
We also release a neural multi-modal retrieval model that can use images or sentences as inputs and retrieves entities in the KG.
Can Wikipedia Categories Improve Masked Language Model Pretraining?
Pretrained language models have obtained impressive results for a large set of natural language understanding tasks.
English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too
Intermediate-task training---fine-tuning a pretrained model on an intermediate task before fine-tuning again on the target task---often improves model performance substantially on language understanding tasks in monolingual English settings.
Ranked #20 on
Zero-Shot Cross-Lingual Transfer
on XTREME
Latent Variable Model for Multi-modal Translation
In this work, we propose to model the interaction between visual and textual features for multi-modal neural machine translation (MMT) through a latent variable model.
Ranked #9 on
Multimodal Machine Translation
on Multi30K
Sentence-Level Multilingual Multi-modal Embedding for Natural Language Processing
We propose a novel discriminative ranking model that learns embeddings from multilingual and multi-modal data, meaning that our model can take advantage of images and descriptions in multiple languages to improve embedding quality.
Incorporating Global Visual Features into Attention-based Neural Machine Translation.
We introduce multi-modal, attention-based neural machine translation (NMT) models which incorporate visual features into different parts of both the encoder and the decoder.
Linguistic realisation as machine translation: Comparing different MT models for AMR-to-text generation
In this paper, we study AMR-to-text generation, framing it as a translation task and comparing two different MT approaches (Phrase-based and Neural MT).
Human Evaluation of Multi-modal Neural Machine Translation: A Case-Study on E-Commerce Listing Titles
Nonetheless, human evaluators ranked translations from a multi-modal NMT model as better than those of a text-only NMT over 88{\%} of the time, which suggests that images do help NMT in this use-case.
Using Images to Improve Machine-Translating E-Commerce Product Listings.
We evaluate our models quantitatively using BLEU and TER and find that (i) additional synthetic data has a general positive impact on text-only and multi-modal NMT models, and that (ii) using a multi-modal NMT model for re-ranking n-best lists improves TER significantly across different n-best list sizes.
Doubly-Attentive Decoder for Multi-modal Neural Machine Translation
We introduce a Multi-modal Neural Machine Translation model in which a doubly-attentive decoder naturally incorporates spatial visual features obtained using pre-trained convolutional neural networks, bridging the gap between image description and translation.
Ranked #11 on
Multimodal Machine Translation
on Multi30K
Multilingual Multi-modal Embeddings for Natural Language Processing
We propose a novel discriminative model that learns embeddings from multilingual and multi-modal data, meaning that our model can take advantage of images and descriptions in multiple languages to improve embedding quality.
Incorporating Global Visual Features into Attention-Based Neural Machine Translation
We introduce multi-modal, attention-based neural machine translation (NMT) models which incorporate visual features into different parts of both the encoder and the decoder.
Ranked #10 on
Multimodal Machine Translation
on Multi30K
Developing a Dataset for Evaluating Approaches for Document Expansion with Images
Motivated by the adage that a {``}picture is worth a thousand words{''} it can be reasoned that automatically enriching the textual content of a document with relevant images can increase the readability of a document.
