Search Results for author:
Found 48 papers, 15 papers with code
A Small but Informed and Diverse Model: The Case of the Multimodal GuessWhat!? Guessing Game
Pre-trained Vision and Language Transformers achieve high performance on downstream tasks due to their ability to transfer representational knowledge accumulated during pretraining on substantial amounts of data.
ACT-Thor: A Controlled Benchmark for Embodied Action Understanding in Simulated Environments
This paves the way for a systematic way of evaluating embodied AI agents that understand grounded actions.
Visually Grounded Follow-up Questions: a Dataset of Spatial Questions Which Require Dialogue History
We call the question that restricts the context: trigger, and we call the spatial question that requires the trigger question to be answered: zoomer.
They Are Not All Alike: Answering Different Spatial Questions Requires Different Grounding Strategies
By analyzing LXMERT errors and its attention mechanisms, we find that our classification helps to gain a better understanding of the skills required to answer different spatial questions.
A MIND for Reasoning: Meta-learning for In-context Deduction
The goal of MIND is to enable models to generalize more effectively to unseen knowledge bases and to systematically apply inference rules.
Playpen: An Environment for Exploring Learning Through Conversational Interaction
Are we running out of learning signal?
All-in-one: Understanding and Generation in Multimodal Reasoning with the MAIA Benchmark
We introduce MAIA (Multimodal AI Assessment), a native-Italian benchmark designed for fine-grained investigation of the reasoning abilities of visual language models on videos.
Triangulating LLM Progress through Benchmarks, Games, and Cognitive Tests
We examine three evaluation paradigms: large question-answering benchmarks (e. g., MMLU and BBH), interactive games (e. g., Signalling Games or Taboo), and cognitive tests (e. g., for working memory or theory of mind).
The Validation Gap: A Mechanistic Analysis of How Language Models Compute Arithmetic but Fail to Validate It
The ability of large language models (LLMs) to validate their output and identify potential errors is crucial for ensuring robustness and reliability.
LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
There is an increasing trend towards evaluating NLP models with LLMs instead of human judgments, raising questions about the validity of these evaluations, as well as their reproducibility in the case of proprietary models.
Learning to Ask Informative Questions: Enhancing LLMs with Preference Optimization and Expected Information Gain
Questions are essential tools for acquiring the necessary information to complete information-seeking tasks.
A Systematic Analysis of Large Language Models as Soft Reasoners: The Case of Syllogistic Inferences
The reasoning abilities of Large Language Models (LLMs) are becoming a central focus of study in NLP.
Looking for Confirmations: An Effective and Human-Like Visual Dialogue Strategy
Inspired by the cognitive literature on information search and cross-situational word learning, we design Confirm-it, a model based on a beam search re-ranking algorithm that guides an effective goal-oriented strategy by asking questions that confirm the model's conjecture about the referent.
``I've Seen Things You People Wouldn't Believe'': Hallucinating Entities in GuessWhat?!
We also analyse where hallucinations tend to occur more often through the dialogue: hallucinations are less frequent in earlier turns, cause a cascade hallucination effect, and are often preceded by negative answers, which have been shown to be harder to ground.
The Interplay of Task Success and Dialogue Quality: An in-depth Evaluation in Task-Oriented Visual Dialogues
When training a model on referential dialogue guessing games, the best model is usually chosen based on its task success.
Overprotective Training Environments Fall Short at Testing Time: Let Models Contribute to Their Own Training
Despite important progress, conversational systems often generate dialogues that sound unnatural to humans.
Be Different to Be Better! A Benchmark to Leverage the Complementarity of Language and Vision
This paper introduces BD2BB, a novel language and vision benchmark that requires multimodal models combine complementary information from the two modalities.
On the role of effective and referring questions in GuessWhat?!
Regarding our first metric, we find that successful dialogues do not have a higher percentage of effective questions for most models.
Effective questions in referential visual dialogue
Task success is the standard metric used to evaluate these systems.
Psycholinguistics meets Continual Learning: Measuring Catastrophic Forgetting in Visual Question Answering
We study the issue of catastrophic forgetting in the context of neural multimodal approaches to Visual Question Answering (VQA).
Quantifiers in a Multimodal World: Hallucinating Vision with Language and Sound
Inspired by the literature on multisensory integration, we develop a computational model to ground quantifiers in perception.
Evaluating the Representational Hub of Language and Vision Models
The multimodal models used in the emerging field at the intersection of computational linguistics and computer vision implement the bottom-up processing of the `Hub and Spoke' architecture proposed in cognitive science to represent how the brain processes and combines multi-sensory inputs.
Beyond task success: A closer look at jointly learning to see, ask, and GuessWhat
We compare our approach to an alternative system which extends the baseline with reinforcement learning.
Grounded Textual Entailment
Capturing semantic relations between sentences, such as entailment, is a long-standing challenge for computational semantics.
Ranked #2 on
Natural Language Inference
on V-SNLI
Ask No More: Deciding when to guess in referential visual dialogue
We make initial steps towards this general goal by augmenting a task-oriented visual dialogue model with a decision-making component that decides whether to ask a follow-up question to identify a target referent in an image, or to stop the conversation to make a guess.
Comparatives, Quantifiers, Proportions: A Multi-Task Model for the Learning of Quantities from Vision
The present work investigates whether different quantification mechanisms (set comparison, vague quantification, and proportional estimation) can be jointly learned from visual scenes by a multi-task computational model.
FOIL it! Find One mismatch between Image and Language caption
In this paper, we aim to understand whether current language and vision (LaVi) models truly grasp the interaction between the two modalities.
Pay Attention to Those Sets! Learning Quantification from Images
We however argue that precisely identifying the composition of the sets is not only beyond current state-of-the-art models but perhaps even detrimental to a task that is most efficiently performed by refining the approximate numerosity estimator of the system.
Be Precise or Fuzzy: Learning the Meaning of Cardinals and Quantifiers from Vision
People can refer to quantities in a visual scene by using either exact cardinals (e. g. one, two, three) or natural language quantifiers (e. g. few, most, all).
The LAMBADA dataset: Word prediction requiring a broad discourse context
We introduce LAMBADA, a dataset to evaluate the capabilities of computational models for text understanding by means of a word prediction task.
Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures
Automatic description generation from natural images is a challenging problem that has recently received a large amount of interest from the computer vision and natural language processing communities.
Unveiling the Dreams of Word Embeddings: Towards Language-Driven Image Generation
We introduce language-driven image generation, the task of generating an image visualizing the semantic contents of a word embedding, e. g., given the word embedding of grasshopper, we generate a natural image of a grasshopper.
A SICK cure for the evaluation of compositional distributional semantic models
Shared and internationally recognized benchmarks are fundamental for the development of any computational system.
