Search Results for author:
Found 61 papers, 27 papers with code
TNG-CLIP:Training-Time Negation Data Generation for Negation Awareness of CLIP
To expand the horizon, we (1) introduce a training-time negation data generation pipeline such that negation captions are generated during the training stage, which only increases 2. 5% extra training time, and (2) we propose the first benchmark, Neg-TtoI, for evaluating text-to-image generation models on prompts containing negation, assessing model's ability to produce semantically accurate images.
Zero-Shot Iterative Formalization and Planning in Partially Observable Environments
Using LLMs not to predict plans but to formalize an environment into the Planning Domain Definition Language (PDDL) has been shown to improve performance and control.
Adjust for Trust: Mitigating Trust-Induced Inappropriate Reliance on AI Assistance
Trust biases how users rely on AI recommendations in AI-assisted decision-making tasks, with low and high levels of trust resulting in increased under- and over-reliance, respectively.
Efficient Evaluation of Multi-Task Robot Policies With Active Experiment Selection
By prioritizing informative trials, our framework reduces the cost of calculating evaluation metrics for robot policies across many tasks.
Better Slow than Sorry: Introducing Positive Friction for Reliable Dialogue Systems
While theories of discourse and cognitive science have long recognized the value of unhurried pacing, recent dialogue research tends to minimize friction in conversational systems.
M3PT: A Transformer for Multimodal, Multi-Party Social Signal Prediction with Person-aware Blockwise Attention
Past work in multi-party interaction tends to build task-specific models for predicting social signals.
Evaluating Creativity and Deception in Large Language Models: A Simulation Framework for Multi-Agent Balderdash
This paper introduces a simulation framework utilizing the game Balderdash to evaluate both the creativity and logical reasoning of LLMs.
The American Sign Language Knowledge Graph: Infusing ASL Models with Linguistic Knowledge
Language models for American Sign Language (ASL) could make language technologies substantially more accessible to those who sign.
Generating Contextually-Relevant Navigation Instructions for Blind and Low Vision People
Navigating unfamiliar environments presents significant challenges for blind and low-vision (BLV) individuals.
When Parts Are Greater Than Sums: Individual LLM Components Can Outperform Full Models
This paper studies in-context learning by decomposing the output of large language models into the individual contributions of attention heads and MLPs (components).
Contrast Sets for Evaluating Language-Guided Robot Policies
We investigate the relationship between experimenter effort to carry out an evaluation and the resulting estimated test performance as well as the insights that can be drawn from performance on perturbed instances.
Language Models can Infer Action Semantics for Symbolic Planners from Environment Feedback
Symbolic planners can discover a sequence of actions from initial to goal states given expert-defined, domain-specific logical action semantics.
TwoStep: Multi-agent Task Planning using Classical Planners and Large Language Models
In contrast to classical planning, large language models (LLMs) directly used for inferring plan steps rarely guarantee execution success, but are capable of leveraging commonsense reasoning to assemble action sequences.
ViSaRL: Visual Reinforcement Learning Guided by Human Saliency
Using ViSaRL to learn visual representations significantly improves the success rate, sample efficiency, and generalization of an RL agent on diverse tasks including DeepMind Control benchmark, robot manipulation in simulation and on a real robot.
Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoning
Selective prediction minimizes incorrect predictions from vision-language models (VLMs) by allowing them to abstain from answering when uncertain.
WinoViz: Probing Visual Properties of Objects Under Different States
We present WinoViz, a text-only evaluation dataset, consisting of 1, 380 examples that probe the reasoning abilities of language models regarding variant visual properties of objects under different contexts or states.
THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation
To realize effective large-scale, real-world robotic applications, we must evaluate how well our robot policies adapt to changes in environmental conditions.
Does VLN Pretraining Work with Nonsensical or Irrelevant Instructions?
Data augmentation via back-translation is common when pretraining Vision-and-Language Navigation (VLN) models, even though the generated instructions are noisy.
Efficient End-to-End Visual Document Understanding with Rationale Distillation
Pre-processing tools, such as optical character recognition (OCR), can map document image inputs to textual tokens, then large language models (LLMs) can reason over text.
Do Localization Methods Actually Localize Memorized Data in LLMs? A Tale of Two Benchmarks
On the other hand, even successful methods identify neurons that are not specific to a single memorized sequence.
Which One? Leveraging Context Between Objects and Multiple Views for Language Grounding
When connecting objects and their language referents in an embodied 3D environment, it is important to note that: (1) an object can be better characterized by leveraging comparative information between itself and other objects, and (2) an object's appearance can vary with camera position.
The Sem-Lex Benchmark: Modeling ASL Signs and Their Phonemes
Sign language recognition and translation technologies have the potential to increase access and inclusion of deaf signing communities, but research progress is bottlenecked by a lack of representative data.
Exploring Strategies for Modeling Sign Language Phonology
Specifically, we explore how learning strategies like multi-task and curriculum learning can leverage mutually useful information between phoneme types to facilitate better modeling of sign language phonemes.
Chain-of-Questions Training with Latent Answers for Robust Multistep Question Answering
We train a language model (LM) to robustly answer multistep questions by generating and answering sub-questions.
I2I: Initializing Adapters with Improvised Knowledge
We propose Improvise to Initialize (I2I), a continual learning algorithm that initializes Adapters for incoming tasks by distilling knowledge from previously-learned tasks' Adapters.
Task-Attentive Transformer Architecture for Continual Learning of Vision-and-Language Tasks Using Knowledge Distillation
The size and the computational load of fine-tuning large-scale pre-trained neural network are becoming two major obstacles in adopting machine learning in many applications.
Multimodal Speech Recognition for Language-Guided Embodied Agents
Benchmarks for language-guided embodied agents typically assume text-based instructions, but deployed agents will encounter spoken instructions.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Improving Sign Recognition with Phonology
We use insights from research on American Sign Language (ASL) phonology to train models for isolated sign language recognition (ISLR), a step towards automatic sign language understanding.
RREx-BoT: Remote Referring Expressions with a Bag of Tricks
Armed with this intuition, using only a generic vision-language scoring model with minor modifications for 3d encoding and operating in an embodied environment, we demonstrate an absolute performance gain of 9. 84% on remote object grounding above state of the art models for REVERIE and of 5. 04% on FAO.
CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigation
Our results on the coarse-grained instruction following task of REVERIE demonstrate the navigational capability of CLIP, surpassing the supervised baseline in terms of both success rate (SR) and success weighted by path length (SPL).
Generalization Differences between End-to-End and Neuro-Symbolic Vision-Language Reasoning Systems
For vision-and-language reasoning tasks, both fully connectionist, end-to-end methods and hybrid, neuro-symbolic methods have achieved high in-distribution performance.
Retrospectives on the Embodied AI Workshop
We present a retrospective on the state of Embodied AI research.
Iterative Vision-and-Language Navigation
We present Iterative Vision-and-Language Navigation (IVLN), a paradigm for evaluating language-guided agents navigating in a persistent environment over time.
ProgPrompt: Generating Situated Robot Task Plans using Large Language Models
To ameliorate that effort, large language models (LLMs) can be used to score potential next actions during task planning, and even generate action sequences directly, given an instruction in natural language with no additional domain information.
VAuLT: Augmenting the Vision-and-Language Transformer for Sentiment Classification on Social Media
VAuLT is an extension of the popular Vision-and-Language Transformer (ViLT), and improves performance on vision-and-language (VL) tasks that involve more complex text inputs than image captions while having minimal impact on training and inference efficiency.
Curriculum Learning for Data-Efficient Vision-Language Alignment
Aligning image and text encoders from scratch using contrastive learning requires large amounts of paired image-text data.
Interactive Learning from Natural Language and Demonstrations using Signal Temporal Logic
In this work, we propose DIALOGUESTL, an interactive approach for learning correct and concise STL formulas from (often) ambiguous NL descriptions.
CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks
Existing CL benchmarks have facilitated research on task adaptation and mitigating "catastrophic forgetting", but are limited to vision-only and language-only tasks.
Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions
A long-term goal of AI research is to build intelligent agents that can communicate with humans in natural language, perceive the environment, and perform real-world tasks.
LUMINOUS: Indoor Scene Generation for Embodied AI Challenges
However, current simulators for Embodied AI (EAI) challenges only provide simulated indoor scenes with a limited number of layouts.
TEACh: Task-driven Embodied Agents that Chat
Robots operating in human spaces must be able to engage in natural language interaction with people, both understanding and executing instructions, and using conversation to resolve ambiguity and recover from mistakes.
Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion
Language-guided robots performing home and office tasks must navigate in and interact with the world.
Language Grounding with 3D Objects
We introduce several CLIP-based models for distinguishing objects and demonstrate that while recent advances in jointly modeling vision and language are useful for robotic language understanding, it is still the case that these image-based models are weaker at understanding the 3D nature of objects -- properties which play a key role in manipulation.
RMM: A Recursive Mental Model for Dialogue Navigation
In this paper, we go beyond instruction following and introduce a two-agent task where one agent navigates and asks questions that a second, guiding agent answers.
The RobotSlang Benchmark: Dialog-guided Robot Localization and Navigation
In each trial, the pair first cooperates to localize the robot on a global map visible to the Commander, then the Driver follows Commander instructions to move the robot to a sequence of target objects.
RMM: A Recursive Mental Model for Dialog Navigation
In this paper, we go beyond instruction following and introduce a two-agent task where one agent navigates and asks questions that a second, guiding agent answers.
Experience Grounds Language
Language understanding research is held back by a failure to relate language to the physical world it describes and to the social interactions it facilitates.
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
We present ALFRED (Action Learning From Realistic Environments and Directives), a benchmark for learning a mapping from natural language instructions and egocentric vision to sequences of actions for household tasks.
Vision-and-Dialog Navigation
To train agents that search an environment for a goal location, we define the Navigation from Dialog History task.
Shifting the Baseline: Single Modality Performance on Visual Navigation \& QA
We demonstrate the surprising strength of unimodal baselines in multimodal domains, and make concrete recommendations for best practices in future research.
Improving Robot Success Detection using Static Object Data
We collect over 13 hours of egocentric manipulation data for training a model to reason about whether a robot successfully placed unseen objects in or on one another.
Interpreting Black Box Models via Hypothesis Testing
In science and medicine, model interpretations may be reported as discoveries of natural phenomena or used to guide patient treatments.
Prospection: Interpretable Plans From Language By Predicting the Future
High-level human instructions often correspond to behaviors with multiple implicit steps.
Improving Grounded Natural Language Understanding through Human-Robot Dialog
Natural language understanding for robotics can require substantial domain- and platform-specific engineering.
Shifting the Baseline: Single Modality Performance on Visual Navigation & QA
We demonstrate the surprising strength of unimodal baselines in multimodal domains, and make concrete recommendations for best practices in future research.
Improving Black-box Speech Recognition using Semantic Parsing
Speech is a natural channel for human-computer interaction in robotics and consumer applications.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+6
Interpretable Low-Dimensional Regression via Data-Adaptive Smoothing
We consider the problem of estimating a regression function in the common situation where the number of features is small, where interpretability of the model is a high priority, and where simple linear or additive models fail to provide adequate performance.
Guiding Interaction Behaviors for Multi-modal Grounded Language Learning
Multi-modal grounded language learning connects language predicates to physical properties of objects in the world.
Integrated Learning of Dialog Strategies and Semantic Parsing
Natural language understanding and dialog management are two integral components of interactive dialog systems.
