Search Results for author:
Found 40 papers, 9 papers with code
A Small but Informed and Diverse Model: The Case of the Multimodal GuessWhat!? Guessing Game
Pre-trained Vision and Language Transformers achieve high performance on downstream tasks due to their ability to transfer representational knowledge accumulated during pretraining on substantial amounts of data.
They Are Not All Alike: Answering Different Spatial Questions Requires Different Grounding Strategies
By analyzing LXMERT errors and its attention mechanisms, we find that our classification helps to gain a better understanding of the skills required to answer different spatial questions.
ACT-Thor: A Controlled Benchmark for Embodied Action Understanding in Simulated Environments
This paves the way for a systematic way of evaluating embodied AI agents that understand grounded actions.
Visually Grounded Follow-up Questions: a Dataset of Spatial Questions Which Require Dialogue History
We call the question that restricts the context: trigger, and we call the spatial question that requires the trigger question to be answered: zoomer.
Looking for Confirmations: An Effective and Human-Like Visual Dialogue Strategy
Inspired by the cognitive literature on information search and cross-situational word learning, we design Confirm-it, a model based on a beam search re-ranking algorithm that guides an effective goal-oriented strategy by asking questions that confirm the model's conjecture about the referent.
``I've Seen Things You People Wouldn't Believe'': Hallucinating Entities in GuessWhat?!
We also analyse where hallucinations tend to occur more often through the dialogue: hallucinations are less frequent in earlier turns, cause a cascade hallucination effect, and are often preceded by negative answers, which have been shown to be harder to ground.
Overprotective Training Environments Fall Short at Testing Time: Let Models Contribute to Their Own Training
Despite important progress, conversational systems often generate dialogues that sound unnatural to humans.
The Interplay of Task Success and Dialogue Quality: An in-depth Evaluation in Task-Oriented Visual Dialogues
When training a model on referential dialogue guessing games, the best model is usually chosen based on its task success.
Be Different to Be Better! A Benchmark to Leverage the Complementarity of Language and Vision
This paper introduces BD2BB, a novel language and vision benchmark that requires multimodal models combine complementary information from the two modalities.
On the role of effective and referring questions in GuessWhat?!
Regarding our first metric, we find that successful dialogues do not have a higher percentage of effective questions for most models.
Effective questions in referential visual dialogue
Task success is the standard metric used to evaluate these systems.
Psycholinguistics meets Continual Learning: Measuring Catastrophic Forgetting in Visual Question Answering
We study the issue of catastrophic forgetting in the context of neural multimodal approaches to Visual Question Answering (VQA).
Quantifiers in a Multimodal World: Hallucinating Vision with Language and Sound
Inspired by the literature on multisensory integration, we develop a computational model to ground quantifiers in perception.
Evaluating the Representational Hub of Language and Vision Models
The multimodal models used in the emerging field at the intersection of computational linguistics and computer vision implement the bottom-up processing of the `Hub and Spoke' architecture proposed in cognitive science to represent how the brain processes and combines multi-sensory inputs.
Beyond task success: A closer look at jointly learning to see, ask, and GuessWhat
We compare our approach to an alternative system which extends the baseline with reinforcement learning.
Grounded Textual Entailment
Capturing semantic relations between sentences, such as entailment, is a long-standing challenge for computational semantics.
Ranked #2 on
Natural Language Inference
on V-SNLI
Ask No More: Deciding when to guess in referential visual dialogue
We make initial steps towards this general goal by augmenting a task-oriented visual dialogue model with a decision-making component that decides whether to ask a follow-up question to identify a target referent in an image, or to stop the conversation to make a guess.
Comparatives, Quantifiers, Proportions: A Multi-Task Model for the Learning of Quantities from Vision
The present work investigates whether different quantification mechanisms (set comparison, vague quantification, and proportional estimation) can be jointly learned from visual scenes by a multi-task computational model.
FOIL it! Find One mismatch between Image and Language caption
In this paper, we aim to understand whether current language and vision (LaVi) models truly grasp the interaction between the two modalities.
Pay Attention to Those Sets! Learning Quantification from Images
We however argue that precisely identifying the composition of the sets is not only beyond current state-of-the-art models but perhaps even detrimental to a task that is most efficiently performed by refining the approximate numerosity estimator of the system.
Be Precise or Fuzzy: Learning the Meaning of Cardinals and Quantifiers from Vision
People can refer to quantities in a visual scene by using either exact cardinals (e. g. one, two, three) or natural language quantifiers (e. g. few, most, all).
The LAMBADA dataset: Word prediction requiring a broad discourse context
We introduce LAMBADA, a dataset to evaluate the capabilities of computational models for text understanding by means of a word prediction task.
Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures
Automatic description generation from natural images is a challenging problem that has recently received a large amount of interest from the computer vision and natural language processing communities.
Unveiling the Dreams of Word Embeddings: Towards Language-Driven Image Generation
We introduce language-driven image generation, the task of generating an image visualizing the semantic contents of a word embedding, e. g., given the word embedding of grasshopper, we generate a natural image of a grasshopper.
A SICK cure for the evaluation of compositional distributional semantic models
Shared and internationally recognized benchmarks are fundamental for the development of any computational system.
