Search Results for author:
Found 54 papers, 32 papers with code
Probing Cross-Modal Representations in Multi-Step Relational Reasoning
We investigate the representations learned by vision and language models in tasks that require relational reasoning.
Sequential Compositional Generalization in Multimodal Models
The rise of large-scale multimodal models has paved the pathway for groundbreaking advances in generative modeling and reasoning, unlocking transformative applications in a variety of complex tasks.
Text Rendering Strategies for Pixel Language Models
Pixel-based language models process text rendered as images, which allows them to handle any script, making them a promising approach to open vocabulary language modelling.
Evaluating Bias and Fairness in Gender-Neutral Pretrained Vision-and-Language Models
We quantify bias amplification in pretraining and after fine-tuning on three families of vision-and-language models.
PHD: Pixel-Based Language Modeling of Historical Documents
We then pre-train our model, PHD, on a combination of synthetic scans and real historical newspapers from the 1700-1900 period.
LMCap: Few-shot Multilingual Image Captioning by Retrieval Augmented Language Model Prompting
Multilingual image captioning has recently been tackled by training with large-scale machine translated data, which is an expensive, noisy, and time-consuming process.
The Role of Data Curation in Image Captioning
Image captioning models are typically trained by treating all samples equally, neglecting to account for mismatched or otherwise difficult data points.
Retrieval-augmented Image Captioning
The encoder in our model jointly processes the image and retrieved captions using a pretrained V&L BERT, while the decoder attends to the multimodal encoder representations, benefiting from the extra textual evidence from the retrieved captions.
Multilingual Multimodal Learning with Machine Translated Text
We call this framework TD-MML: Translated Data for Multilingual Multimodal Learning, and it can be applied to any multimodal dataset and model.
Zero-Shot Cross-Lingual Image-to-Text Retrieval
Zero-Shot Cross-Lingual Text-to-Image Retrieval
+3
An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classification
Non-hierarchical sparse attention Transformer-based models, such as Longformer and Big Bird, are popular approaches to working with long documents.
SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentation
Recent advances in image captioning have focused on scaling the data and model size, substantially increasing the cost of pre-training and finetuning.
Language Modelling with Pixels
We pretrain the 86M parameter PIXEL model on the same English data as BERT and evaluate on syntactic and semantic tasks in typologically diverse languages, including various non-Latin scripts.
Ranked #1 on
Named Entity Recognition (NER)
on MasakhaNER
Revisiting Transformer-based Models for Long Document Classification
The recent literature in text classification is biased towards short text sequences (e. g., sentences or paragraphs).
IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages
Our benchmark enables the evaluation of multilingual multimodal models for transfer learning, not only in a zero-shot setting, but also in newly defined few-shot learning setups.
Visually Grounded Reasoning across Languages and Cultures
The design of widespread vision-and-language datasets and pre-trained encoders directly adopts, or draws inspiration from, the concepts and images of ImageNet.
Ranked #1 on
Zero-Shot Cross-Lingual Transfer
on MaRVL
MDAPT: Multilingual Domain Adaptive Pretraining in a Single Model
Domain adaptive pretraining, i. e. the continued unsupervised pretraining of a language model on domain-specific text, improves the modelling of text for downstream tasks within the domain.
Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers
Models that have learned to construct cross-modal representations using both modalities are expected to perform worse when inputs are missing from a modality.
The Role of Syntactic Planning in Compositional Image Captioning
Image captioning has focused on generalizing to images drawn from the same distribution as the training set, and not to the more challenging problem of generalizing to different distributions of images.
Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs
Large-scale pretraining and task-specific fine-tuning is now the standard methodology for many tasks in computer vision and natural language processing.
Multimodal Speech Recognition with Unstructured Audio Masking
Our experiments on the Flickr 8K Audio Captions Corpus show that multimodal ASR can generalize to recover different types of masked words in this unstructured masking setting.
Textual Supervision for Visually Grounded Spoken Language Understanding
Visually-grounded models of spoken language understanding extract semantic information directly from speech, without relying on transcriptions.
Fine-Grained Grounding for Multimodal Speech Recognition
In experiments on the Flickr8K Audio Captions Corpus, we find that our model improves over approaches that use global visual features, that the proposals enable the model to recover entities and other related words, such as adjectives, and that improvements are due to the model's ability to localize the correct proposals.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
On Forgetting to Cite Older Papers: An Analysis of the ACL Anthology
The field of natural language processing is experiencing a period of unprecedented growth, and with it a surge of published papers.
CompGuessWhat?!: A Multi-task Evaluation Framework for Grounded Language Learning
To remedy this, we present GROLLA, an evaluation framework for Grounded Language Learning with Attributes with three sub-tasks: 1) Goal-oriented evaluation; 2) Object attribute prediction evaluation; and 3) Zero-shot evaluation.
The Sensitivity of Language Models and Humans to Winograd Schema Perturbations
Large-scale pretrained language models are the major driving force behind recent improvements in performance on the Winograd Schema Challenge, a widely employed test of common sense reasoning ability.
Multimodal Machine Translation through Visuals and Speech
Multimodal machine translation involves drawing information from more than one modality, based on the assumption that the additional modalities will contain useful alternative views of the input data.
Ranked #4 on
Multimodal Machine Translation
on Multi30K
Bootstrapping Disjoint Datasets for Multilingual Multimodal Representation Learning
However, we do find that using an external machine translation model to generate the synthetic data sets results in better performance.
Adversarial Removal of Demographic Attributes Revisited
Elazar and Goldberg (2018) showed that protected attributes can be extracted from the representations of a debiased neural network for mention detection at above-chance levels, by evaluating a diagnostic classifier on a held-out subsample of the data it was trained on.
Understanding the Effect of Textual Adversaries in Multimodal Machine Translation
It is assumed that multimodal machine translation systems are better than text-only systems at translating phrases that have a direct correspondence in the image.
Compositional Generalization in Image Captioning
Image captioning models are usually evaluated on their ability to describe a held-out set of images, not on their ability to generalize to unseen concepts.
Cross-lingual Visual Verb Sense Disambiguation
We extend this line of work to the more challenging task of cross-lingual verb sense disambiguation, introducing the MultiSense dataset of 9, 504 images annotated with English, German, and Spanish verbs.
Talking about other people: an endless range of possibilities
This taxonomy serves as a reference point to think about how other people should be described, and can be used to classify and compute statistics about labels applied to people.
How2: A Large-scale Dataset for Multimodal Language Understanding
In this paper, we introduce How2, a multimodal collection of instructional videos with English subtitles and crowdsourced Portuguese translations.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Adversarial Evaluation of Multimodal Machine Translation
The promise of combining language and vision in multimodal machine translation is that systems will produce better translations by leveraging the image data.
Findings of the Third Shared Task on Multimodal Machine Translation
In this task a source sentence in English is supplemented by an image and participating systems are required to generate a translation for such a sentence into German, French or Czech.
Lessons learned in multilingual grounded language learning
Recent work has shown how to learn better visual-semantic embeddings by leveraging image descriptions in more than one language.
Measuring the Diversity of Automatic Image Descriptions
Automatic image description systems typically produce generic sentences that only make use of a small subset of the vocabulary available to them.
Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description
The multilingual image description task was changed such that at test time, only the image is given.
Cross-linguistic differences and similarities in image descriptions
Automatic image description systems are commonly trained and evaluated on large image description datasets.
Imagination improves Multimodal Translation
We decompose multimodal translation into two sub-tasks: learning to translate and learning visually grounded representations.
Room for improvement in automatic image description: an error analysis
In recent years we have seen rapid and significant progress in automatic image description but what are the open problems in this area?
Pragmatic factors in image description: the case of negations
We provide a qualitative analysis of the descriptions containing negations (no, not, n't, nobody, etc) in the Flickr30K corpus, and a categorization of negation uses.
Multi30K: Multilingual English-German Image Descriptions
We introduce the Multi30K dataset to stimulate multilingual multimodal research.
1 Million Captioned Dutch Newspaper Images
Images naturally appear alongside text in a wide variety of media, such as books, magazines, newspapers, and in online articles.
A Corpus of Images and Text in Online News
The corpus consists of JSON-LD files with the following data about each article: the original URL of the article on the news publisher{'}s website, the date of publication, the headline of the article, the URL of the image displayed with the article (if any), and the caption of that image.
Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures
Automatic description generation from natural images is a challenging problem that has recently received a large amount of interest from the computer vision and natural language processing communities.
Multilingual Image Description with Neural Sequence Models
In this paper we present an approach to multi-language image description bringing together insights from neural machine translation and neural image description.
