Search Results for author:
Found 74 papers, 25 papers with code
Demonstrating EMMA: Embodied MultiModal Agent for Language-guided Action Execution in 3D Simulated Environments
We demonstrate EMMA, an embodied multimodal agent which has been developed for the Alexa Prize SimBot challenge.
ConvAbuse: Data, Analysis, and Benchmarks for Nuanced Detection in Conversational AI
We find that the distribution of abuse is vastly different compared to other commonly used datasets, with more sexually tinted aggression towards the virtual persona of these systems.
Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design
Over the last several years, end-to-end neural conversational agents have vastly improved their ability to carry unrestricted, open-domain conversations with humans.
Conversational Assistants and Gender Stereotypes: Public Perceptions and Desiderata for Voice Personas
We then outline a multi-disciplinary project of how we plan to address the complex question of gender and stereotyping in digital assistants.
What happens if you treat ordinal ratings as interval data? Human evaluations in NLP are even more under-powered than you think
Previous work has shown that human evaluations in NLP are notoriously under-powered.
SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems
We then empirically assess the extent to which current tools can measure these effects and current systems display them.
Twenty Years of Confusion in Human Evaluation: NLG Needs Evaluation Sheets and Standardised Definitions
Human assessment remains the most trusted form of evaluation in NLG, but highly diverse approaches and a proliferation of different quality criteria used by researchers make it difficult to compare results and draw conclusions across papers, with adverse implications for meta-evaluation and reproducibility.
Multitask Multimodal Prompted Training for Interactive Embodied Task Completion
Interactive and embodied tasks pose at least two fundamental challenges to existing Vision & Language (VL) models, including 1) grounding language in trajectories of actions and observations, and 2) referential disambiguation.
Sociotechnical Safety Evaluation of Generative AI Systems
First, we propose a three-layered framework that takes a structured, sociotechnical approach to evaluating these risks.
FurChat: An Embodied Conversational Agent using LLMs, Combining Open and Closed-Domain Dialogue with Facial Expressions
We demonstrate an embodied conversational agent that can function as a receptionist and generate a mixture of open and closed-domain dialogue along with facial expressions, by using a large language model (LLM) to develop an engaging conversation.
Understanding Counterspeech for Online Harm Mitigation
Counterspeech offers direct rebuttals to hateful speech by challenging perpetrators of hate and showing support to targets of abuse.
The Dangers of trusting Stochastic Parrots: Faithfulness and Trust in Open-domain Conversational Question Answering
Large language models are known to produce output which sounds fluent and convincing, but is also often wrong, e. g. "unfaithful" with respect to a rationale as retrieved from a knowledge base.
Conversational Question Answering
Open-Domain Question Answering
Mirages: On Anthropomorphism in Dialogue Systems
In this paper, we discuss the linguistic factors that contribute to the anthropomorphism of dialogue systems and the harms that can arise, including reinforcing gender stereotypes and notions of acceptable language.
iLab at SemEval-2023 Task 11 Le-Wi-Di: Modelling Disagreement or Modelling Perspectives?
There are two competing approaches for modelling annotator disagreement: distributional soft-labelling approaches (which aim to capture the level of disagreement) or modelling perspectives of individual annotators or groups thereof.
ANTONIO: Towards a Systematic Method of Generating NLP Benchmarks for Verification
In particular, many known neural network verification methods that work for computer vision and other numeric datasets do not work for NLP.
Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP
We report our efforts in identifying a set of previous human evaluations in NLP that would be suitable for a coordinated study examining what makes human evaluations in NLP more/less reproducible.
SemEval-2023 Task 11: Learning With Disagreements (LeWiDi)
We report on the second LeWiDi shared task, which differs from the first edition in three crucial respects: (i) it focuses entirely on NLP, instead of both NLP and computer vision tasks in its first edition; (ii) it focuses on subjective tasks, instead of covering different types of disagreements-as training with aggregated labels for subjective NLP tasks is a particularly obvious misrepresentation of the data; and (iii) for the evaluation, we concentrate on soft approaches to evaluation.
Quality-agnostic Image Captioning to Safely Assist People with Vision Impairment
In this paper, we propose a quality-agnostic framework to improve the performance and robustness of image captioning models for visually impaired people.
Consistency is Key: Disentangling Label Variation in Natural Language Processing with Intra-Annotator Agreement
We commonly use agreement measures to assess the utility of judgements made by human annotators in Natural Language Processing (NLP) tasks.
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions.
Going for GOAL: A Resource for Grounded Football Commentaries
As the course of a game is unpredictable, so are commentaries, which makes them a unique resource to investigate dynamic language grounding.
Risk-graded Safety for Handling Medical Queries in Conversational AI
While individual crowdworkers may be unreliable at grading the seriousness of the prompts, their aggregated labels tend to agree with professional opinion to a greater extent on identifying the medical queries and recognising the risk types posed by the responses.
Task Formulation Matters When Learning Continually: A Case Study in Visual Question Answering
We first propose three plausible task formulations and demonstrate their impact on the performance of continual learning algorithms.
Adversarial Robustness of Visual Dialog
This study is the first to investigate the robustness of visually grounded dialog models towards textual attacks.
Why Robust Natural Language Understanding is a Challenge
With the proliferation of Deep Machine Learning into real-life applications, a particular property of this technology has been brought to attention: robustness Neural Networks notoriously present low robustness and can be highly sensitive to small input perturbations.
Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challenges
This is a report on the NSF Future Directions Workshop on Automatic Evaluation of Dialog.
MiRANews: Dataset and Benchmarks for Multi-Resource-Assisted News Summarization
We show via data analysis that it's not only the models which are to blame: more than 27% of facts mentioned in the gold summaries of MiRANews are better grounded on assisting documents than in the main source articles.
ConvAbuse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational AI
We find that the distribution of abuse is vastly different compared to other commonly used datasets, with more sexually tinted aggression towards the virtual persona of these systems.
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling
Over the last several years, end-to-end neural conversational agents have vastly improved in their ability to carry a chit-chat conversation with humans.
AGGGEN: Ordering and Aggregating while Generating
We present AGGGEN (pronounced 'again'), a data-to-text model which re-introduces two explicit sentence planning stages into neural data-to-text systems: input ordering and input aggregation.
Alexa, Google, Siri: What are Your Pronouns? Gender and Anthropomorphism in the Design and Perception of Conversational Assistants
Technology companies have produced varied responses to concerns about the effects of the design of their conversational AI systems.
OTTers: One-turn Topic Transitions for Open-Domain Dialogue
Mixed initiative in open-domain dialogue requires a system to pro-actively introduce new topics.
SLURP: A Spoken Language Understanding Resource Package
Spoken Language Understanding infers semantic meaning directly from audio data, and thus promises to reduce error propagation and misunderstandings in end-user applications.
Ranked #3 on
Slot Filling
on SLURP
(using extra training data)
Fact-based Content Weighting for Evaluating Abstractive Summarisation
Abstractive summarisation is notoriously hard to evaluate since standard word-overlap-based metrics are insufficient.
History for Visual Dialog: Do we really need it?
Visual Dialog involves "understanding" the dialog history (what has been discussed previously) and the current question (what is asked), in addition to grounding information in the image, to generate the correct response.
Semantic Noise Matters for Neural Natural Language Generation
Neural natural language generation (NNLG) systems are known for their pathological outputs, i. e. generating text which is unrelated to the input specification.
Ranked #3 on
Data-to-Text Generation
on Cleaned E2E NLG Challenge
Automatic Quality Estimation for Natural Language Generation: Ranting (Jointly Rating and Ranking)
We present a recurrent neural network based system for automatic quality estimation of natural language generation (NLG) outputs, which jointly learns to assign numerical ratings to individual outputs and to provide pairwise rankings of two different outputs.
A Crowd-based Evaluation of Abuse Response Strategies in Conversational Agents
How should conversational agents respond to verbal abuse through the user?
User Evaluation of a Multi-dimensional Statistical Dialogue System
We present the first complete spoken dialogue system driven by a multi-dimensional statistical dialogue manager.
Benchmarking Natural Language Understanding Services for building Conversational Agents
We have recently seen the emergence of several publicly available Natural Language Understanding (NLU) toolkits, which map user utterances to structured, but more abstract, Dialogue Act (DA) or Intent specifications, while making this process accessible to the lay developer.
Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge
Introducing novel automatic and human metrics, we compare 62 systems submitted by 17 institutions, covering a wide range of approaches, including machine learning architectures -- with the majority implementing sequence-to-sequence models (seq2seq) -- as well as systems based on grammatical rules and templates.
Improving Context Modelling in Multimodal Dialogue Generation
In this work, we investigate the task of textual response generation in a multimodal task-oriented dialogue system.
A Knowledge-Grounded Multimodal Search-Based Conversational Agent
Multimodal search-based dialogue is a challenging new task: It extends visually grounded question answering systems into multi-turn conversations with access to an external database.
Findings of the E2E NLG Challenge
This paper summarises the experimental setup and results of the first shared task on end-to-end (E2E) natural language generation (NLG) in spoken dialogue systems.
Ranked #4 on
Data-to-Text Generation
on E2E NLG Challenge
Better Conversations by Modeling, Filtering, and Optimizing for Coherence and Diversity
We present three enhancements to existing encoder-decoder models for open-domain conversational agents, aimed at effectively modeling coherence and promoting output diversity: (1) We introduce a measure of coherence as the GloVe embedding similarity between the dialogue context and the generated response, (2) we filter our training corpora based on the measure of coherence to obtain topically coherent and lexically diverse context-response pairs, (3) we then train a response generator using a conditional variational autoencoder model that incorporates the measure of coherence as a latent variable and uses a context gate to guarantee topical consistency with the context and promote lexical diversity.
Better Conversations by Modeling,Filtering,and Optimizing for Coherence and Diversity
We present three enhancements to existing encoder-decoder models for open-domain conversational agents, aimed at effectively modeling coherence and promoting output diversity: (1) We introduce a measure of coherence as the GloVe embedding similarity between the dialogue context and the generated response, (2) we filter our training corpora based on the measure of coherence to obtain topically coherent and lexically diverse context-response pairs, (3) we then train a response generator using a conditional variational autoencoder model that incorporates the measure of coherence as a latent variable and uses a context gate to guarantee topical consistency with the context and promote lexical diversity.
\#MeToo Alexa: How Conversational Systems Respond to Sexual Harassment
In this article, we establish how current state-of-the-art conversational systems react to inappropriate requests, such as bullying and sexual harassment on the part of the user, by collecting and analysing the novel {\#}MeTooAlexa corpus.
Towards Learning Transferable Conversational Skills using Multi-dimensional Dialogue Modelling
Recent statistical approaches have improved the robustness and scalability of spoken dialogue systems.
RankME: Reliable Human Ratings for Natural Language Generation
Human evaluation for natural language generation (NLG) often suffers from inconsistent user ratings.
An Ensemble Model with Ranking for Social Dialogue
Open-domain social dialogue is one of the long-standing goals of Artificial Intelligence.
A Review of Evaluation Techniques for Social Dialogue Systems
In contrast with goal-oriented dialogue, social dialogue has no clear measure of task success.
Referenceless Quality Estimation for Natural Language Generation
Traditional automatic evaluation measures for natural language generation (NLG) use costly human-authored references to estimate the quality of a system output.
Why We Need New Evaluation Metrics for NLG
The majority of NLG evaluation relies on automatic metrics, such as BLEU .
Data-driven Natural Language Generation: Paving the Road to Success
We argue that there are currently two major bottlenecks to the commercial use of statistical machine learning approaches for natural language generation (NLG): (a) The lack of reliable automatic evaluation metrics for NLG, and (b) The scarcity of high quality in-domain corpora.
The E2E Dataset: New Challenges For End-to-End Generation
This paper describes the E2E data, a new dataset for training end-to-end, data-driven natural language generation systems in the restaurant domain, which is ten times bigger than existing, frequently used datasets in this area.
Crowd-sourcing NLG Data: Pictures Elicit Better Data
Recent advances in corpus-based Natural Language Generation (NLG) hold the promise of being easily portable across domains, but require costly training data, consisting of meaning representations (MRs) paired with Natural Language (NL) utterances.
Natural Language Generation as Planning under Uncertainty Using Reinforcement Learning
We present and evaluate a new model for Natural Language Generation (NLG) in Spoken Dialogue Systems, based on statistical planning, given noisy feedback from the current generation context (e. g. a user and a surface realiser).
Natural Language Generation enhances human decision-making with uncertain information
Decision-making is often dependent on uncertain data, e. g. data associated with confidence scores or probabilities.
The REAL Corpus: A Crowd-Sourced Corpus of Human Generated and Evaluated Spatial References to Real-World Urban Scenes
Our interest is in people{'}s capacity to efficiently and effectively describe geographic objects in urban scenes.
An Arabic Twitter Corpus for Subjectivity and Sentiment Analysis
We present a newly collected data set of 8, 868 gold-standard annotated Arabic feeds.
