Search Results for author:
Found 44 papers, 10 papers with code
Where Am I and Where Should I Go? Grounding Positional and Directional Labels in a Disoriented Human Balancing Task
In this paper, we present an approach toward grounding linguistic positional and directional labels directly to human motions in the course of a disoriented balancing task in a multi-axis rotational device.
A Generalized Method for Automated Multilingual Loanword Detection
Loanwords are words incorporated from one language into another without translation.
Embodied Multimodal Agents to Bridge the Understanding Gap
In this paper we argue that embodied multimodal agents, i. e., avatars, can play an important role in moving natural language processing toward “deep understanding.” Fully-featured interactive agents, model encounters between two “people,” but a language-only agent has little environmental and situational awareness.
The VoxWorld Platform for Multimodal Embodied Agents
We present a five-year retrospective on the development of the VoxWorld platform, first introduced as a multimodal platform for modeling motion language, that has evolved into a platform for rapidly building and deploying embodied agents with contextual and situational awareness, capable of interacting with humans in multiple modalities, and exploring their environments.
Phonetic, Semantic, and Articulatory Features in Assamese-Bengali Cognate Detection
In this paper, we propose a method to detect if words in two similar languages, Assamese and Bengali, are cognates.
TRACE: Real-Time Multimodal Common Ground Tracking in Situated Collaborative Dialogues
We present TRACE, a novel system for live *common ground* tracking in situated collaborative tasks.
Speech Is Not Enough: Interpreting Nonverbal Indicators of Common Knowledge and Engagement
Our goal is to develop an AI Partner that can provide support for group problem solving and social dynamics.
Any Other Thoughts, Hedgehog? Linking Deliberation Chains in Collaborative Dialogues
Our framework jointly models probing and causal utterances and the links between them, and we evaluate on two challenging collaborative task datasets: the Weights Task and DeliData.
Simultaneous Reward Distillation and Preference Learning: Get You a Language Model Who Can Do Both
Traditional RLHF-based LLM alignment methods explicitly maximize the expected rewards from a separate reward model.
Combating Spatial Disorientation in a Dynamic Self-Stabilization Task Using AI Assistants
These were used in a co-performance study with 20 new human subjects performing a version of the VIP task with degraded spatial information.
Metacognitive AI: Framework and the Case for a Neurosymbolic Approach
Metacognition is the concept of reasoning about an agent's own internal processes and was originally introduced in the field of developmental psychology.
Computational Thought Experiments for a More Rigorous Philosophy and Science of the Mind
We offer philosophical motivations for a method we call Virtual World Cognitive Science (VW CogSci), in which researchers use virtual embodied agents that are embedded in virtual worlds to explore questions in the field of Cognitive Science.
Multimodal Cross-Document Event Coreference Resolution Using Linear Semantic Transfer and Mixed-Modality Ensembles
We establish three methods that incorporate images and text for coreference: 1) a standard fused model with finetuning, 2) a novel linear mapping method without finetuning and 3) an ensembling approach based on splitting mention pairs by semantic and discourse-level difficulty.
Okay, Let's Do This! Modeling Event Coreference with Generated Rationales and Knowledge Distillation
In NLP, Event Coreference Resolution (ECR) is the task of connecting event clusters that refer to the same underlying real-life event, usually via neural systems.
Cross-Lingual Transfer Robustness to Lower-Resource Languages on Adversarial Datasets
Multilingual Language Models (MLLMs) exhibit robust cross-lingual transfer capabilities, or the ability to leverage information acquired in a source language and apply it to a target language.
Common Ground Tracking in Multimodal Dialogue
Within Dialogue Modeling research in AI and NLP, considerable attention has been spent on ``dialogue state tracking'' (DST), which is the ability to update the representations of the speaker's needs at each turn in the dialogue by taking into account the past dialogue moves and history.
Exploring Failure Cases in Multimodal Reasoning About Physical Dynamics
In this paper, we present an exploration of LLMs' abilities to problem solve with physical reasoning in situated environments.
2∗n is better than n^2: Decomposing Event Coreference Resolution into Two Tractable Problems
Event Coreference Resolution (ECR) is the task of linking mentions of the same event either within or across documents.
How Good is the Model in Model-in-the-loop Event Coreference Resolution Annotation?
Annotating cross-document event coreference links is a time-consuming and cognitively demanding task that can compromise annotation quality and efficiency.
How Good is Automatic Segmentation as a Multimodal Discourse Annotation Aid?
We (1) manually transcribe utterances in a dataset of triads collaboratively solving a problem involving dialogue and physical object manipulation, (2) annotate collaborative moves according to these gold-standard transcripts, and then (3) apply these annotations to utterances that have been automatically segmented using toolkits from Google and OpenAI's Whisper.
AxomiyaBERTa: A Phonologically-aware Transformer Model for Assamese
Moreover, we show that AxomiyaBERTa can leverage phonological signals for even more challenging tasks, such as a novel cross-document coreference task on a translated version of the ECB+ corpus, where we present a new SOTA result for an LRL.
Grounding and Distinguishing Conceptual Vocabulary Through Similarity Learning in Embodied Simulations
We present a novel method for using agent experiences gathered through an embodied simulation to ground contextualized word vectors to object representations.
An Abstract Specification of VoxML as an Annotation Language
VoxML is a modeling language used to map natural language expressions into real-time visualizations using commonsense semantic knowledge of objects and events.
$2 * n$ is better than $n^2$: Decomposing Event Coreference Resolution into Two Tractable Problems
Event Coreference Resolution (ECR) is the task of linking mentions of the same event either within or across documents.
Detecting and Accommodating Novel Types and Concepts in an Embodied Simulation Environment
In this paper, we present methods for two types of metacognitive tasks in an AI system: rapidly expanding a neural classification model to accommodate a new category of object, and recognizing when a novel object type is observed instead of misclassifying the observation as a known class.
Exploiting Embodied Simulation to Detect Novel Object Classes Through Interaction
In this paper we present a novel method for a naive agent to detect novel objects it encounters in an interaction.
Neurosymbolic AI for Situated Language Understanding
In recent years, data-intensive AI, particularly the domain of natural language processing and understanding, has seen significant progress driven by the advent of large datasets and deep neural networks that have sidelined more classic AI approaches to the field.
Situated Multimodal Control of a Mobile Robot: Navigation through a Virtual Environment
We present a new interface for controlling a navigation robot in novel environments using coordinated gesture and language.
A Formal Analysis of Multimodal Referring Strategies Under Common Ground
In this paper, we present an analysis of computationally generated mixed-modality definite referring expressions using combinations of gesture and linguistic descriptions.
Multimodal Continuation-style Architectures for Human-Robot Interaction
We present an architecture for integrating real-time, multimodal input into a computational agent's contextual model.
Generating a Novel Dataset of Multimodal Referring Expressions
Referring expressions and definite descriptions of objects in space exploit information both about object characteristics and locations.
Situational Grounding within Multimodal Simulations
In this paper, we argue that simulation platforms enable a novel type of embodied spatial reasoning, one facilitated by a formal model of object and event semantics that renders the continuous quantitative search space of an open-world, real-time environment tractable.
Combining Deep Learning and Qualitative Spatial Reasoning to Learn Complex Structures from Sparse Examples with Noise
We present a novel approach to introducing new spatial structures to an AI agent, combining deep learning over qualitative spatial relations with various heuristic search algorithms.
Every Object Tells a Story
Most work within the computational event modeling community has tended to focus on the interpretation and ordering of events that are associated with verbs and event nominals in linguistic expressions.
Building Multimodal Simulations for Natural Language
Simulation and automatic visualization of events from natural language descriptions and supplementary modalities, such as gestures, allows humans to use their native capabilities as linguistic and visual interpreters to collaborate on tasks with an artificial agent or to put semantic intuitions to the test in an environment where user and agent share a common context. In previous work (Pustejovsky and Krishnaswamy, 2014; Pustejovsky, 2013a), we introduced a method for modeling natural language expressions within a 3D simulation environment built on top of the game development platform Unity (Goldstone, 2009).
The Development of Multimodal Lexical Resources
Human communication is a multimodal activity, involving not only speech and written expressions, but intonation, images, gestures, visual clues, and the interpretation of actions through perception.
VoxSim: A Visual Platform for Modeling Motion Language
Much existing work in text-to-scene generation focuses on generating static scenes.
Generating Simulations of Motion Events from Verbal Descriptions
The generated simulations act as a conceptual "debugger" for the semantics of different motion verbs: that is, by testing for consistency and informativeness in the model, simulations expose the presuppositions associated with linguistic expressions and their compositions.
ECAT: Event Capture Annotation Tool
This paper introduces the Event Capture Annotation Tool (ECAT), a user-friendly, open-source interface tool for annotating events and their participants in video, capable of extracting the 3D positions and orientations of objects in video captured by Microsoft's Kinect(R) hardware.
VoxML: A Visualization Modeling Language
We present the specification for a modeling language, VoxML, which encodes semantic knowledge of real-world objects represented as three-dimensional models, and of events and attributes related to and enacted over these objects.
Multimodal Semantic Simulations of Linguistically Underspecified Motion Events
In this paper, we describe a system for generating three-dimensional visual simulations of natural language motion expressions.
