Search Results for author:
Found 18 papers, 6 papers with code
Combine to Describe: Evaluating Compositional Generalization in Image Captioning
Compositionality – the ability to combine simpler concepts to understand & generate arbitrarily more complex conceptual structures – has long been thought to be the cornerstone of human language capacity.
Demonstrating EMMA: Embodied MultiModal Agent for Language-guided Action Execution in 3D Simulated Environments
We demonstrate EMMA, an embodied multimodal agent which has been developed for the Alexa Prize SimBot challenge.
ACT-Thor: A Controlled Benchmark for Embodied Action Understanding in Simulated Environments
This paves the way for a systematic way of evaluating embodied AI agents that understand grounded actions.
Dialogue Act and Slot Recognition in Italian Complex Dialogues
Since the advent of Transformer-based, pretrained language models (LM) such as BERT, Natural Language Understanding (NLU) components in the form of Dialogue Act Recognition (DAR) and Slot Recognition (SR) for dialogue systems have become both more accurate and easier to create for specific application domains.
Lost in Space: Probing Fine-grained Spatial Understanding in Vision and Language Resamplers
In this paper, we use \textit{diagnostic classifiers} to measure the extent to which the visual prompt produced by the resampler encodes spatial information.
PIXAR: Auto-Regressive Language Modeling in Pixel Space
However, these pixel-based LLMs are limited to discriminative tasks (e. g., classification) and, similar to BERT, cannot be used to generate text.
Is Feedback All You Need? Leveraging Natural Language Feedback in Goal-Conditioned Reinforcement Learning
Despite numerous successes, the field of reinforcement learning (RL) remains far from matching the impressive generalisation power of human behaviour learning.
Visually Grounded Language Learning: a review of language games, datasets, tasks, and models
Our analysis of the literature provides evidence that future work should be focusing on interactive games where communication in Natural Language is important to resolve ambiguities about object referents and action plans and that physical embodiment is essential to understand the semantics of situations and events.
Multitask Multimodal Prompted Training for Interactive Embodied Task Completion
Interactive and embodied tasks pose at least two fundamental challenges to existing Vision & Language (VL) models, including 1) grounding language in trajectories of actions and observations, and 2) referential disambiguation.
'What are you referring to?' Evaluating the Ability of Multi-Modal Dialogue Models to Process Clarificational Exchanges
Referential ambiguities arise in dialogue when a referring expression does not uniquely identify the intended referent for the addressee.
Going for GOAL: A Resource for Grounded Football Commentaries
As the course of a game is unpredictable, so are commentaries, which makes them a unique resource to investigate dynamic language grounding.
Task Formulation Matters When Learning Continually: A Case Study in Visual Question Answering
We first propose three plausible task formulations and demonstrate their impact on the performance of continual learning algorithms.
Exploring Multi-Modal Representations for Ambiguity Detection & Coreference Resolution in the SIMMC 2.0 Challenge
Anaphoric expressions, such as pronouns and referential descriptions, are situated with respect to the linguistic context of prior turns, as well as, the immediate visual environment.
Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion
Language-guided robots performing home and office tasks must navigate in and interact with the world.
An Empirical Study on the Generalization Power of Neural Representations Learned via Visual Guessing Games
Guessing games are a prototypical instance of the "learning by interacting" paradigm.
Imagining Grounded Conceptual Representations from Perceptual Information in Situated Guessing Games
However, as shown by Suglia et al. (2020), existing models fail to learn truly multi-modal representations, relying instead on gold category labels for objects in the scene both at training and inference time.
CompGuessWhat?!: A Multi-task Evaluation Framework for Grounded Language Learning
To remedy this, we present GROLLA, an evaluation framework for Grounded Language Learning with Attributes with three sub-tasks: 1) Goal-oriented evaluation; 2) Object attribute prediction evaluation; and 3) Zero-shot evaluation.
Iterative Multi-document Neural Attention for Multiple Answer Prediction
People have information needs of varying complexity, which can be solved by an intelligent agent able to answer questions formulated in a proper way, eventually considering user context and preferences.
