Search Results for author:
Found 68 papers, 11 papers with code
Conversational Agents for Intelligent Buildings
We will demonstrate a deployed conversational AI system that acts as a host of a smart-building on a university campus.
Demonstrating EMMA: Embodied MultiModal Agent for Language-guided Action Execution in 3D Simulated Environments
We demonstrate EMMA, an embodied multimodal agent which has been developed for the Alexa Prize SimBot challenge.
A Visually-Aware Conversational Robot Receptionist
Socially Assistive Robots (SARs) have the potential to play an increasingly important role in a variety of contexts including healthcare, but most existing systems have very limited interactive capabilities.
The Spoon Is in the Sink: Assisting Visually Impaired People in the Kitchen
Questions about the spatial relations between these objects are particularly helpful to visually impaired people, and our system output more usable answers than other state of the art end-to-end VQA systems.
Shaking Up VLMs: Comparing Transformers and Structured State Space Models for Vision & Language Modeling
We explore two hypotheses to explain this phenomenon: 1) the effect of task-agnostic visual encoding on the updates of the hidden states, and 2) the difficulty in performing visual grounding from the perspective of in-context multimodal retrieval.
AlanaVLM: A Multimodal Embodied AI Foundation Model for Egocentric Video Understanding
First, we introduce the Egocentric Video Understanding Dataset (EVUD) for training VLMs on video captioning and question answering tasks specific to egocentric videos.
Lost in Space: Probing Fine-grained Spatial Understanding in Vision and Language Resamplers
In this paper, we use \textit{diagnostic classifiers} to measure the extent to which the visual prompt produced by the resampler encodes spatial information.
Socially Pertinent Robots in Gerontological Healthcare
Despite the many recent achievements in developing and deploying social robotics, there are still many underexplored environments and applications for which systematic evaluation of such systems by end-users is necessary.
NLP Verification: Towards a General Methodology for Certifying Robustness
We propose a number of practical NLP methods that can help to quantify the effects of the embedding gap; and in particular we propose the metric of falsifiability of semantic subspaces as another fundamental metric to be reported as part of the NLP verification pipeline.
Visually Grounded Language Learning: a review of language games, datasets, tasks, and models
Our analysis of the literature provides evidence that future work should be focusing on interactive games where communication in Natural Language is important to resolve ambiguities about object referents and action plans and that physical embodiment is essential to understand the semantics of situations and events.
Multitask Multimodal Prompted Training for Interactive Embodied Task Completion
Interactive and embodied tasks pose at least two fundamental challenges to existing Vision & Language (VL) models, including 1) grounding language in trajectories of actions and observations, and 2) referential disambiguation.
Detecting agreement in multi-party dialogue: evaluating speaker diarisation versus a procedural baseline to enhance user engagement
Conversational agents participating in multi-party interactions face significant challenges in dialogue state tracking, since the identity of the speaker adds significant contextual meaning.
Detecting Agreement in Multi-party Conversational AI
Today, conversational systems are expected to handle conversations in multi-party settings, especially within Socially Assistive Robots (SARs).
Multi-party Goal Tracking with LLMs: Comparing Pre-training, Fine-tuning, and Prompt Engineering
This paper evaluates the extent to which current Large Language Models (LLMs) can capture task-oriented multi-party conversations (MPCs).
FurChat: An Embodied Conversational Agent using LLMs, Combining Open and Closed-Domain Dialogue with Facial Expressions
We demonstrate an embodied conversational agent that can function as a receptionist and generate a mixture of open and closed-domain dialogue along with facial expressions, by using a large language model (LLM) to develop an engaging conversation.
SimpleMTOD: A Simple Language Model for Multimodal Task-Oriented Dialogue with Symbolic Scene Representation
SimpleMTOD is a simple language model which recasts several sub-tasks in multimodal task-oriented dialogues as sequence prediction tasks.
An Empirical Study on the Generalization Power of Neural Representations Learned via Visual Guessing Games
Guessing games are a prototypical instance of the "learning by interacting" paradigm.
Encoding Syntactic Constituency Paths for Frame-Semantic Parsing with Graph Convolutional Networks
We study the problem of integrating syntactic information from constituency trees into a neural model in Frame-semantic parsing sub-tasks, namely Target Identification (TI), FrameIdentification (FI), and Semantic Role Labeling (SRL).
Imagining Grounded Conceptual Representations from Perceptual Information in Situated Guessing Games
However, as shown by Suglia et al. (2020), existing models fail to learn truly multi-modal representations, relying instead on gold category labels for objects in the scene both at training and inference time.
CompGuessWhat?!: A Multi-task Evaluation Framework for Grounded Language Learning
To remedy this, we present GROLLA, an evaluation framework for Grounded Language Learning with Attributes with three sub-tasks: 1) Goal-oriented evaluation; 2) Object attribute prediction evaluation; and 3) Zero-shot evaluation.
Data-Efficient Goal-Oriented Conversation with Dialogue Knowledge Transfer Networks
Our main dataset is the Stanford Multi-Domain dialogue corpus.
Hierarchical Multi-Task Natural Language Understanding for Cross-domain Conversational AI: HERMIT NLU
We present a new neural architecture for wide-coverage Natural Language Understanding in Spoken Dialogue Systems.
MuMMER: Socially Intelligent Human-Robot Interaction in Public Spaces
In the EU-funded MuMMER project, we have developed a social robot designed to interact naturally and flexibly with users in public spaces such as a shopping mall.
Petri Net Machines for Human-Agent Interaction
Smart speakers and robots become ever more prevalent in our daily lives.
Few-Shot Dialogue Generation Without Annotated Data: A Transfer Learning Approach
Learning with minimal data is one of the key challenges in the development of practical, production-ready goal-oriented dialogue systems.
Neural Response Ranking for Social Conversation: A Data-Efficient Approach
Using a dataset of real conversations collected in the 2017 Alexa Prize challenge, we developed a neural ranker for selecting 'good' system responses to user utterances, i. e. responses which are likely to lead to long and engaging conversations.
Multi-Task Learning for Domain-General Spoken Disfluency Detection in Dialogue Systems
To test the model's generalisation potential, we evaluate the same model on the bAbI+ dataset, without any additional training.
An Ensemble Model with Ranking for Social Dialogue
Open-domain social dialogue is one of the long-standing goals of Artificial Intelligence.
Learning how to learn: an adaptive dialogue agent for incrementally learning visually grounded word meanings
We present an optimised multi-modal dialogue agent for interactive learning of visually grounded word meanings from a human tutor, trained on real human-human tutoring data.
Training an adaptive dialogue policy for interactive learning of visually grounded word meanings
We present a multi-modal dialogue system for interactive learning of perceptually grounded word meanings from a human tutor.
Challenging Neural Dialogue Models with Natural Data: Memory Networks Fail on Incremental Phenomena
Results show that the semantic accuracy of the MemN2N model drops drastically; and that although it is in principle able to learn to process the constructions in bAbI+, it needs an impractical amount of training data to do so.
Bootstrapping incremental dialogue systems from minimal data: the generalisation power of dialogue grammars
Our experiments show that our model can process 74% of the Facebook AI bAbI dataset even when trained on only 0. 13% of the data (5 dialogues).
VOILA: An Optimised Dialogue System for Interactively Learning Visually-Grounded Word Meanings (Demonstration System)
We present VOILA: an optimised, multi-modal dialogue agent for interactive learning of visually grounded word meanings from a human user.
Sympathy Begins with a Smile, Intelligence Begins with a Word: Use of Multimodal Features in Spoken Human-Robot Interaction
We find that happiness in the user's recognised facial expression strongly correlates with likeability of a robot, while dialogue-related features (such as number of human turns or number of sentences per robot utterance) correlate with perceiving a robot as intelligent.
Evaluating Persuasion Strategies and Deep Reinforcement Learning methods for Negotiation Dialogue agents
In this paper we present a comparative evaluation of various negotiation strategies within an online version of the game {``}Settlers of Catan{''}.
Bootstrapping incremental dialogue systems: using linguistic knowledge to learn from minimal data
We present a method for inducing new dialogue systems from very small amounts of unannotated dialogue data, showing how word-level exploration using Reinforcement Learning (RL), combined with an incremental and semantic grammar - Dynamic Syntax (DS) - allows systems to discover, generate, and understand many new dialogue variants.
Crowd-sourcing NLG Data: Pictures Elicit Better Data
Recent advances in corpus-based Natural Language Generation (NLG) hold the promise of being easily portable across domains, but require costly training data, consisting of meaning representations (MRs) paired with Natural Language (NL) utterances.
Natural Language Generation as Planning under Uncertainty Using Reinforcement Learning
We present and evaluate a new model for Natural Language Generation (NLG) in Spoken Dialogue Systems, based on statistical planning, given noisy feedback from the current generation context (e. g. a user and a surface realiser).
Natural Language Generation enhances human decision-making with uncertain information
Decision-making is often dependent on uncertain data, e. g. data associated with confidence scores or probabilities.
Strategic Dialogue Management via Deep Reinforcement Learning
This paper describes a successful application of Deep Reinforcement Learning (DRL) for training intelligent agents with strategic conversational skills, in a situated dialogue setting.
On the Linear Belief Compression of POMDPs: A re-examination of current methods
Belief compression improves the tractability of large-scale partially observable Markov decision processes (POMDPs) by finding projections from high-dimensional belief space onto low-dimensional approximations, where solving to obtain action selection policies requires fewer computations.
