Search Results for author:
Found 68 papers, 30 papers with code
SPRING: GPT-4 Out-performs RL Algorithms by Studying Papers and Reasoning
Finally, we show the potential of games as a test bed for LLMs.
Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agents
We propose the Plan, Eliminate, and Track (PET) framework.
Computational Language Acquisition with Theory of Mind
We also find some evidence that increasing task difficulty in the training process results in more fluent and precise utterances in evaluation.
The Framework Tax: Disparities Between Inference Efficiency in Research and Deployment
Increased focus on the deployment of machine learning systems has led to rapid improvements in hardware accelerator performance and neural network model efficiency.
EXCALIBUR: Encouraging and Evaluating Embodied Exploration
Humans constantly explore and learn about their environment out of curiosity, gather information, and update their models of the world.
Self-Supervised Object Goal Navigation with In-Situ Finetuning
In contrast, our LocCon shows the most robust transfer in the real world among the set of models we compare to, and that the real-world performance of all models can be further improved with self-supervised LocCon in-situ training.
EvEntS ReaLM: Event Reasoning of Entity States via Language Models
This paper investigates models of event implications.
Retrospectives on the Embodied AI Workshop
We present a retrospective on the state of Embodied AI research.
Don't Copy the Teacher: Data and Model Challenges in Embodied Dialogue
We provide empirical comparisons of metrics, analysis of three models, and make suggestions for how the field might best progress.
Transformers are Adaptable Task Planners
Every home is different, and every person likes things done in their particular way.
On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization
Combining the visual modality with pretrained language models has been surprisingly effective for simple descriptive tasks such as image captioning.
Training Vision-Language Transformers from Captions Alone
Vision-Language Transformers can be learned without human labels (e. g. class labels, bounding boxes, etc).
HEAR: Holistic Evaluation of Audio Representations
The aim of the HEAR benchmark is to develop a general-purpose audio representation that provides a strong basis for learning in a wide variety of tasks and scenarios.
KAT: A Knowledge Augmented Transformer for Vision-and-Language
The primary focus of recent work with largescale transformers has been on optimizing the amount of information packed into the model's parameters.
A Framework for Learning to Request Rich and Contextually Useful Information from Humans
We show that the agent can take advantage of different types of information depending on the context, and analyze the benefits and challenges of learning the assistance-requesting policy when the assistant can recursively decompose tasks into subtasks.
FILM: Following Instructions in Language with Modular Methods
In contrast, we propose a modular method with structured representations that (1) builds a semantic map of the scene and (2) performs exploration with a semantic search policy, to achieve the natural language goal.
Learning When and What to Ask: a Hierarchical Reinforcement Learning Framework
Results on a simulated human-assisted navigation problem demonstrate the effectiveness of our framework: aided with an interaction policy learned by our method, a navigation policy achieves up to a 7× improvement in task success rate compared to performing tasks only by itself.
Hierarchical Reinforcement Learning
reinforcement-learning
+1
Symmetric Machine Theory of Mind
Theory of mind (ToM), the ability to understand others' thoughts and desires, is a cornerstone of human intelligence.
Shaped Rewards Bias Emergent Language
The second is shaped rewards which are designed specifically to make the task easier to learn by introducing biases in the learning process.
Dependency Induction Through the Lens of Visual Perception
Our experiments find that concreteness is a strong indicator for learning dependency grammars, improving the direct attachment score (DAS) by over 50\% as compared to state-of-the-art models trained on pure text.
WebQA: Multihop and Multimodal QA
Scaling Visual Question Answering (VQA) to the open-domain and multi-hop nature of web searches, requires fundamental advances in visual representation learning, knowledge aggregation, and language generation.
TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment
This is motivated by the observation that for a video-text pair, the content words in the text, such as nouns and verbs, are more likely to be aligned with the visual contents in the video than the function words.
Ranked #3 on
Zero-Shot Video Retrieval
on YouCook2
Language Grounding with 3D Objects
We introduce several CLIP-based models for distinguishing objects and demonstrate that while recent advances in jointly modeling vision and language are useful for robotic language understanding, it is still the case that these image-based models are weaker at understanding the 3D nature of objects -- properties which play a key role in manipulation.
Few-shot Language Coordination by Modeling Theory of Mind
Positive results from our experiments hint at the importance of explicitly modeling communication as a socio-pragmatic progress.
Grounding 'Grounding' in NLP
And finally, (3) How to advance our current definition to bridge the gap with Cognitive Science?
Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Models
Numerous works have analyzed biases in vision and pre-trained language models individually - however, less attention has been paid to how these biases interact in multimodal settings.
An Empirical Study on the Generalization Power of Neural Representations Learned via Visual Guessing Games
Guessing games are a prototypical instance of the "learning by interacting" paradigm.
BUTLER: Building Understanding in TextWorld via Language for Embodied Reasoning
ALFWorld enables the creation of a new BUTLER agent whose abstract knowledge, learned in TextWorld, corresponds directly to concrete, visually grounded actions.
Token-Level Contrast for Video and Language Alignment
Building video and language understanding models requires grounding linguistic concepts and video contents into a shared space.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
Guided by a set of hypotheses, the framework studies how to transform various pre-existing knowledge resources into a form that is most effective for pre-training models.
Imagining Grounded Conceptual Representations from Perceptual Information in Situated Guessing Games
However, as shown by Suglia et al. (2020), existing models fail to learn truly multi-modal representations, relying instead on gold category labels for objects in the scene both at training and inference time.
RMM: A Recursive Mental Model for Dialogue Navigation
In this paper, we go beyond instruction following and introduce a two-agent task where one agent navigates and asks questions that a second, guiding agent answers.
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
ALFWorld enables the creation of a new BUTLER agent whose abstract knowledge, learned in TextWorld, corresponds directly to concrete, visually grounded actions.
The Return of Lexical Dependencies: Neural Lexicalized PCFGs
In this paper we demonstrate that $\textit{context free grammar (CFG) based methods for grammar induction benefit from modeling lexical dependencies}$.
RMM: A Recursive Mental Model for Dialog Navigation
In this paper, we go beyond instruction following and introduce a two-agent task where one agent navigates and asks questions that a second, guiding agent answers.
A Benchmark for Structured Procedural Knowledge Extraction from Cooking Videos
Watching instructional videos are often used to learn about procedures.
Experience Grounds Language
Language understanding research is held back by a failure to relate language to the physical world it describes and to the social interactions it facilitates.
Multi-View Learning for Vision-and-Language Navigation
Learning to navigate in a visual environment following natural language instructions is a challenging task because natural language instructions are highly variable, ambiguous, and under-specified.
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
We present ALFRED (Action Learning From Realistic Environments and Directives), a benchmark for learning a mapping from natural language instructions and egocentric vision to sequences of actions for household tasks.
PIQA: Reasoning about Physical Commonsense in Natural Language
Questions requiring this kind of physical commonsense pose a challenge to today's natural language understanding systems.
Natural Language Understanding
Physical Commonsense Reasoning
+1
Robust Navigation with Language Pretraining and Stochastic Sampling
Core to the vision-and-language navigation (VLN) challenge is building robust instruction representations and action decoding schemes, which can generalize well to previously unseen instructions and environments.
Benchmarking Hierarchical Script Knowledge
Understanding procedural language requires reasoning about both hierarchical and temporal relations between events.
Shifting the Baseline: Single Modality Performance on Visual Navigation \& QA
We demonstrate the surprising strength of unimodal baselines in multimodal domains, and make concrete recommendations for best practices in future research.
Defending Against Neural Fake News
We find that best current discriminators can classify neural fake news from real, human-written, news with 73% accuracy, assuming access to a moderate level of training data.
Ranked #2 on
Fake News Detection
on Grover-Mega
HellaSwag: Can a Machine Really Finish Your Sentence?
In this paper, we show that commonsense inference still proves difficult for even state-of-the-art models, by presenting HellaSwag, a new challenge dataset.
Improving Robot Success Detection using Static Object Data
We collect over 13 hours of egocentric manipulation data for training a model to reason about whether a robot successfully placed unseen objects in or on one another.
Prospection: Interpretable Plans From Language By Predicting the Future
High-level human instructions often correspond to behaviors with multiple implicit steps.
Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigation
We present the Frontier Aware Search with backTracking (FAST) Navigator, a general framework for action decoding, that achieves state-of-the-art results on the Room-to-Room (R2R) Vision-and-Language navigation challenge of Anderson et.
Ranked #3 on
Vision-Language Navigation
on Room2Room
Character-based Surprisal as a Model of Reading Difficulty in the Presence of Error
Also, transpositions are more difficult than misspellings, and a high error rate increases difficulty for all words, including correct ones.
From Recognition to Cognition: Visual Commonsense Reasoning
While this task is easy for humans, it is tremendously difficult for today's vision systems, requiring higher-order cognition and commonsense reasoning about the world.
Multiple-choice
Multiple Choice Question Answering (MCQA)
+1
Early Fusion for Goal Directed Robotic Vision
Building perceptual systems for robotics which perform well under tight computational budgets requires novel architectures which rethink the traditional computer vision pipeline.
Shifting the Baseline: Single Modality Performance on Visual Navigation & QA
We demonstrate the surprising strength of unimodal baselines in multimodal domains, and make concrete recommendations for best practices in future research.
SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference
Given a partial description like "she opened the hood of the car," humans can reason about the situation and anticipate what might come next ("then, she examined the engine").
Ranked #4 on
Common Sense Reasoning
on SWAG
Inducing Grammars with and for Neural Machine Translation
To address both of these issues we introduce a model that simultaneously translates while inducing dependency trees.
Balancing Shared Autonomy with Human-Robot Communication
Robotic agents that share autonomy with a human should leverage human domain knowledge and account for their preferences when completing a task.
CHALET: Cornell House Agent Learning Environment
We present CHALET, a 3D house simulator with support for navigation and manipulation.
Learning Interpretable Spatial Operations in a Rich 3D Blocks World
In this paper, we study the problem of mapping natural language instructions to complex spatial actions in a 3D blocks world.
Synthetic and Natural Noise Both Break Neural Machine Translation
Character-based neural machine translation (NMT) models alleviate out-of-vocabulary issues, learn morphology, and move us closer to completely end-to-end translation systems.
Natural Language Inference from Multiple Premises
We define a novel textual entailment task that requires inference over multiple premise sentences.
Evaluating Induced CCG Parsers on Grounded Semantic Parsing
We compare the effectiveness of four different syntactic CCG parsers for a semantic slot-filling task to explore how much syntactic supervision is required for downstream semantic analysis.
Unsupervised Neural Hidden Markov Models
In this work, we present the first results for neuralizing an Unsupervised Hidden Markov Model.
An HDP Model for Inducing Combinatory Categorial Grammars
We introduce a novel nonparametric Bayesian model for the induction of Combinatory Categorial Grammars from POS-tagged text.
