Search Results for author:
Found 98 papers, 22 papers with code
From “Before” to “After”: Generating Natural Language Instructions from Image Pairs in a Simple Visual Domain
We find that it is, and investigate several simple baselines, taking these from the related task of image captioning.
New or Old? Exploring How Pre-Trained Language Models Represent Discourse Entities
Recent research shows that pre-trained language models, built to generate text conditioned on some context, learn to encode syntactic knowledge to a certain degree.
Incremental Unit Networks for Multimodal, Fine-grained Information State Representation
We offer a fine-grained information state annotation scheme that follows directly from the Incremental Unit abstract model of dialogue processing when used within a multimodal, co-located, interactive setting.
Annotating anaphoric phenomena in situated dialogue
With this paper, we intend to start a discussion on the annotation of referential phenomena in situated dialogue.
Reference and coreference in situated dialogue
We argue that there is still significant room for corpora that increase the complexity of both visual and linguistic domains and which capture different varieties of perceptual and conversational contexts.
Anaphoric Phenomena in Situated dialog: A First Round of Annotations
We present a first release of 500 documents from the multimodal corpus Tell-me-more (Ilinykh et al., 2019) annotated with coreference information according to the ARRAU guidelines (Poesio et al., 2021).
Playpen: An Environment for Exploring Learning Through Conversational Interaction
Are we running out of learning signal?
Triangulating LLM Progress through Benchmarks, Games, and Cognitive Tests
We examine three evaluation paradigms: large question-answering benchmarks (e. g., MMLU and BBH), interactive games (e. g., Signalling Games or Taboo), and cognitive tests (e. g., for working memory or theory of mind).
Ad-hoc Concept Forming in the Game Codenames as a Means for Evaluating Large Language Models
This study utilizes the game Codenames as a benchmarking tool to evaluate large language models (LLMs) with respect to specific linguistic and cognitive skills.
Plant in Cupboard, Orange on Table, Book on Shelf. Benchmarking Practical Reasoning and Situation Modelling in a Text-Simulated Situated Environment
Large language models (LLMs) have risen to prominence as 'chatbots' for users to interact via natural language.
Incremental Dialogue Management: Survey, Discussion, and Implications for HRI
Efforts towards endowing robots with the ability to speak have benefited from recent advancements in NLP, in particular large language models.
Towards No-Code Programming of Cobots: Experiments with Code Synthesis by Large Code Models for Conversational Programming
While there has been a lot of research recently on robots in household environments, at the present time, most robots in existence can be found on shop floors, and most interactions between humans and robots happen there.
The Unreasonable Ineffectiveness of Nucleus Sampling on Mitigating Text Memorization
This work analyses the text memorization behavior of large language models (LLMs) when subjected to nucleus sampling.
LLMs as Function Approximators: Terminology, Taxonomy, and Questions for Evaluation
Natural Language Processing has moved rather quickly from modelling specific tasks to taking more general pre-trained models and fine-tuning them for specific tasks, to a point where we now have what appear to be inherently generalist models.
LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
There is an increasing trend towards evaluating NLP models with LLMs instead of human judgments, raising questions about the validity of these evaluations, as well as their reproducibility in the case of proprietary models.
Retrieval-Augmented Code Generation for Situated Action Generation: A Case Study on Minecraft
In the Minecraft Collaborative Building Task, two players collaborate: an Architect (A) provides instructions to a Builder (B) to assemble a specified structure using 3D blocks.
Using Game Play to Investigate Multimodal and Conversational Grounding in Large Multimodal Models
On further analysis, we find that the exceptional deep captioning capabilities of the largest models drive some of the performance.
A Dialogue Game for Eliciting Balanced Collaboration
Collaboration is an integral part of human dialogue.
clembench-2024: A Challenging, Dynamic, Complementary, Multilingual Benchmark and Underlying Flexible Framework for LLMs as Multi-Action Agents
In this paper, we take one of the proposed frameworks for setting up such game-play environments, and further test its usefulness as an evaluation instrument, along a number of dimensions: We show that it can easily keep up with new developments while avoiding data contamination, we show that the tests implemented within it are not yet saturated (human performance is substantially higher than that of even the best models), and we show that it lends itself to investigating additional questions, such as the impact of the prompting language on performance.
It Couldn't Help But Overhear: On the Limits of Modelling Meta-Communicative Grounding Acts with Supervised Learning
Active participation in a conversation is key to building common ground, since understanding is jointly tailored by producers and recipients.
Sharing the Cost of Success: A Game for Evaluating and Learning Collaborative Multi-Agent Instruction Giving and Following Policies
In collaborative goal-oriented settings, the participants are not only interested in achieving a successful outcome, but do also implicitly negotiate the effort they put into the interaction (by adapting to each other).
When Only Time Will Tell: Interpreting How Transformers Process Local Ambiguities Through the Lens of Restart-Incrementality
Incremental models that process sentences one token at a time will sometimes encounter points where more than one interpretation is possible.
Learning Communication Policies for Different Follower Behaviors in a Collaborative Reference Game
Albrecht and Stone (2018) state that modeling of changing behaviors remains an open problem "due to the essentially unconstrained nature of what other agents may do".
Taking Action Towards Graceful Interaction: The Effects of Performing Actions on Modelling Policies for Instruction Clarification Requests
Clarification requests are a mechanism to help solve communication problems, e. g. due to ambiguity or underspecification, in instruction-following interactions.
On General Language Understanding
Natural Language Processing prides itself to be an empirically-minded, if not outright empiricist field, and yet lately it seems to get itself into essentialist debates on issues of meaning and measurement ("Do Large Language Models Understand Language, And If So, How Much?").
Revising with a Backward Glance: Regressions and Skips during Reading as Cognitive Signals for Revision Policies in Incremental Processing
In NLP, incremental processors produce output in instalments, based on incoming prefixes of the linguistic input.
Neural Conversation Models and How to Rein Them in: A Survey of Failures and Fixes
In this survey, we interpret Grice's maxims of cooperative conversation from the perspective of this specific research area and systematize the literature under the aspect of what makes a contribution appropriate: A neural conversation model has to be fluent, informative, consistent, coherent, and follow social norms.
The Road to Quality is Paved with Good Revisions: A Detailed Evaluation Methodology for Revision Policies in Incremental Sequence Labelling
Incremental dialogue model components produce a sequence of output prefixes based on incoming input.
"Are you telling me to put glasses on the dog?'' Content-Grounded Annotation of Instruction Clarification Requests in the CoDraw Dataset
Instruction Clarification Requests are a mechanism to solve communication problems, which is very functional in instruction-following interactions.
Pento-DIARef: A Diagnostic Dataset for Learning the Incremental Algorithm for Referring Expression Generation from Examples
NLP tasks are typically defined extensionally through datasets containing example instantiations (e. g., pairs of image i and text t), but motivated intensionally through capabilities invoked in verbal descriptions of the task (e. g., "t is a description of i, for which the content of i needs to be recognised and understood").
Images in Language Space: Exploring the Suitability of Large Language Models for Vision & Language Tasks
Specifically, we investigate the performance of open-source, open-access language models against GPT-3 on five vision-language tasks when given textually-encoded visual information.
Clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agents
Recent work has proposed a methodology for the systematic evaluation of "Situated Language Understanding Agents"-agents that operate in rich linguistic and non-linguistic contexts-through testing them in carefully constructed interactive settings.
Yes, this Way! Learning to Ground Referring Expressions into Actions with Intra-episodic Feedback from Supportive Teachers
The ability to pick up on language signals in an ongoing interaction is crucial for future machine learning models to collaborate and interact with humans naturally.
TAPIR: Learning Adaptive Revision for Incremental Natural Language Understanding with a Two-Pass Model
RNNs are fast but monotonic (cannot correct earlier output, which can be necessary in incremental processing).
Dialogue Games for Benchmarking Language Understanding: Motivation, Taxonomy, Strategy
I argue that such tests need to be complemented with tests of language use embedded in a practice, to arrive at a more comprehensive evaluation of "artificial language understanding".
Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the CoDraw Dataset
In visual instruction-following dialogue games, players can engage in repair mechanisms in face of an ambiguous or underspecified instruction that cannot be fully mapped to actions in the world.
What A Situated Language-Using Agent Must be Able to Do: A Top-Down Analysis
Even in our increasingly text-intensive times, the primary site of language use is situated, co-present interaction.
Norm Participation Grounds Language
The striking recent advances in eliciting seemingly meaningful language behaviour from language-only machine learning models have only made more apparent, through the surfacing of clear limitations, the need to go beyond the language-only mode and to ground these models "in the world".
Can Visual Dialogue Models Do Scorekeeping? Exploring How Dialogue Representations Incrementally Encode Shared Knowledge
Our conclusion is that the ability to make the distinction between shared and privately known statements along the dialogue is moderately present in the analysed models, but not always incrementally consistent, which may partially be due to the limited need for grounding interactions in the original task.
The slurk Interaction Server Framework: Better Data for Better Dialog Models
This paper presents the slurk software, a lightweight interaction server for setting up dialog data collections and running experiments.
Towards Incremental Transformers: An Empirical Analysis of Transformer Models for Incremental NLU
In this work, we examine the feasibility of LT for incremental NLU in English.
Space Efficient Context Encoding for Non-Task-Oriented Dialogue Generation with Graph Attention Transformer
To improve the coherence and knowledge retrieval capabilities of non-task-oriented dialogue systems, recent Transformer-based models aim to integrate fixed background context.
Is Incoherence Surprising? Targeted Evaluation of Coherence Prediction from Language Models
Coherent discourse is distinguished from a mere collection of utterances by the satisfaction of a diverse set of constraints, for example choice of expression, logical relation between denoted events, and implicit compatibility with world-knowledge.
Incremental Processing in the Age of Non-Incremental Encoders: An Empirical Assessment of Bidirectional Models for Incremental NLU
While humans process language incrementally, the best language encoders currently used in NLP do not.
An Overview of Natural Language State Representation for Reinforcement Learning
A suitable state representation is a fundamental part of the learning process in Reinforcement Learning.
Targeting the Benchmark: On Methodology in Current Natural Language Processing Research
It has become a common pattern in our field: One group introduces a language task, exemplified by a dataset, which they argue is challenging enough to serve as a benchmark.
A Corpus of Controlled Opinionated and Knowledgeable Movie Discussions for Training Neural Conversation Models
Fully data driven Chatbots for non-goal oriented dialogues are known to suffer from inconsistent behaviour across their turns, stemming from a general difficulty in controlling parameters like their assumed background personality and knowledge of facts.
Can Neural Image Captioning be Controlled via Forced Attention?
Learned dynamic weighting of the conditioning signal (attention) has been shown to improve neural language generation in a variety of settings.
Tell Me More: A Dataset of Visual Scene Description Sequences
We present a dataset consisting of what we call image description sequences, which are multi-sentence descriptions of the contents of an image.
From Explainability to Explanation: Using a Dialogue Setting to Elicit Annotations with Justifications
Despite recent attempts in the field of explainable AI to go beyond black box prediction models, typically already the training data for supervised machine learning is collected in a manner that treats the annotator as a {``}black box{''}, the internal workings of which remains unobserved.
Grounded Agreement Games: Emphasizing Conversational Grounding in Visual Dialogue Settings
Where early work on dialogue in Computational Linguistics put much emphasis on dialogue structure and its relation to the mental states of the dialogue participants (e. g., Allen 1979, Grosz & Sidner 1986), current work mostly reduces dialogue to the task of producing at any one time a next utterance; e. g. in neural chatbot or Visual Dialogue settings.
Language Tasks and Language Games: On Methodology in Current Natural Language Processing Research
"This paper introduces a new task and a new dataset", "we improve the state of the art in X by Y" -- it is rare to find a current natural language processing paper (or AI paper more generally) that does not contain such statements.
MeetUp! A Corpus of Joint Activity Dialogues in a Visual Environment
Building computer systems that can converse about their visual environment is one of the oldest concerns of research in Artificial Intelligence and Computational Linguistics (see, for example, Winograd's 1972 SHRDLU system).
Know What You Don't Know: Modeling a Pragmatic Speaker that Refers to Objects of Unknown Categories
Zero-shot learning in Language & Vision is the task of correctly labelling (or naming) objects of novel categories.
Natural Language Semantics With Pictures: Some Language \& Vision Datasets and Potential Uses for Computational Semantics
Propelling, and propelled by, the {``}deep learning revolution{''}, recent years have seen the introduction of ever larger corpora of images annotated with natural language expressions.
Natural Language Semantics With Pictures: Some Language & Vision Datasets and Potential Uses for Computational Semantics
Propelling, and propelled by, the "deep learning revolution", recent years have seen the introduction of ever larger corpora of images annotated with natural language expressions.
Decoding Strategies for Neural Referring Expression Generation
In this work, we assess decoding strategies for referring expression generation with neural models.
The Task Matters: Comparing Image Captioning and Task-Based Dialogical Image Description
Image captioning models are typically trained on data that is collected from people who are asked to describe an image, without being given any further task context.
Being data-driven is not enough: Revisiting interactive instruction giving as a challenge for NLG
Modeling traditional NLG tasks with data-driven techniques has been a major focus of research in NLG in the past decade.
Draw and Tell: Multimodal Descriptions Outperform Verbal- or Sketch-Only Descriptions in an Image Retrieval Task
While language conveys meaning largely symbolically, actual communication acts typically contain iconic elements as well: People gesture while they speak, or may even draw sketches while explaining something.
Natural Language Informs the Interpretation of Iconic Gestures: A Computational Approach
When giving descriptions, speakers often signify object shape or size with hand gestures.
Refer-iTTS: A System for Referring in Spoken Installments to Objects in Real-World Images
Current referring expression generation systems mostly deliver their output as one-shot, written expressions.
Deriving continous grounded meaning representations from referentially structured multimodal contexts
Corpora of referring expressions paired with their visual referents are a good source for learning word meanings directly grounded in visual representations.
Beyond On-hold Messages: Conversational Time-buying in Task-oriented Dialogue
A common convention in graphical user interfaces is to indicate a {``}wait state{''}, for example while a program is preparing a response, through a changed cursor state or a progress bar.
Obtaining referential word meanings from visual and distributional information: Experiments on object naming
We present a model that learns individual predictors for object names that link visual and distributional aspects of word meaning during training.
Is this a Child, a Girl or a Car? Exploring the Contribution of Distributional Similarity to Learning Referential Word Meanings
There has recently been a lot of work trying to use images of referents of words for improving vector space meaning representations derived from text.
Grounding Language by Continuous Observation of Instruction Following
Grounded semantics is typically learnt from utterance-level meaning representations (e. g., successful database retrievals, denoted objects in images, moves in a game).
Joint, Incremental Disfluency Detection and Utterance Segmentation from Speech
We present the joint task of incremental disfluency detection and utterance segmentation and a simple deep learning system which performs it on transcripts and ASR results.
How to Address Smart Homes with a Social Robot? A Multi-modal Corpus of User Interactions with an Intelligent Environment
In order to explore intuitive verbal and non-verbal interfaces in smart environments we recorded user interactions with an intelligent apartment.
DUEL: A Multi-lingual Multimodal Dialogue Corpus for Disfluency, Exclamations and Laughter
We present the DUEL corpus, consisting of 24 hours of natural, face-to-face, loosely task-directed dialogue in German, French and Mandarin Chinese.
PentoRef: A Corpus of Spoken References in Task-oriented Dialogues
PentoRef is a corpus of task-oriented dialogues collected in systematically manipulated settings.
Resolving References to Objects in Photographs using the Words-As-Classifiers Model
A common use of language is to refer to visually present objects.
