Search Results for author:
Found 83 papers, 38 papers with code
An HDP Model for Inducing Combinatory Categorial Grammars
We introduce a novel nonparametric Bayesian model for the induction of Combinatory Categorial Grammars from POS-tagged text.
Unsupervised Neural Hidden Markov Models
In this work, we present the first results for neuralizing an Unsupervised Hidden Markov Model.
Evaluating Induced CCG Parsers on Grounded Semantic Parsing
We compare the effectiveness of four different syntactic CCG parsers for a semantic slot-filling task to explore how much syntactic supervision is required for downstream semantic analysis.
Natural Language Inference from Multiple Premises
We define a novel textual entailment task that requires inference over multiple premise sentences.
Synthetic and Natural Noise Both Break Neural Machine Translation
Character-based neural machine translation (NMT) models alleviate out-of-vocabulary issues, learn morphology, and move us closer to completely end-to-end translation systems.
Learning Interpretable Spatial Operations in a Rich 3D Blocks World
In this paper, we study the problem of mapping natural language instructions to complex spatial actions in a 3D blocks world.
CHALET: Cornell House Agent Learning Environment
We present CHALET, a 3D house simulator with support for navigation and manipulation.
Balancing Shared Autonomy with Human-Robot Communication
Robotic agents that share autonomy with a human should leverage human domain knowledge and account for their preferences when completing a task.
Inducing Grammars with and for Neural Machine Translation
To address both of these issues we introduce a model that simultaneously translates while inducing dependency trees.
SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference
Given a partial description like "she opened the hood of the car," humans can reason about the situation and anticipate what might come next ("then, she examined the engine").
Ranked #4 on
Common Sense Reasoning
on SWAG
Shifting the Baseline: Single Modality Performance on Visual Navigation & QA
We demonstrate the surprising strength of unimodal baselines in multimodal domains, and make concrete recommendations for best practices in future research.
Early Fusion for Goal Directed Robotic Vision
Building perceptual systems for robotics which perform well under tight computational budgets requires novel architectures which rethink the traditional computer vision pipeline.
From Recognition to Cognition: Visual Commonsense Reasoning
While this task is easy for humans, it is tremendously difficult for today's vision systems, requiring higher-order cognition and commonsense reasoning about the world.
Multiple-choice
Multiple Choice Question Answering (MCQA)
+1
Character-based Surprisal as a Model of Reading Difficulty in the Presence of Error
Also, transpositions are more difficult than misspellings, and a high error rate increases difficulty for all words, including correct ones.
Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigation
We present the Frontier Aware Search with backTracking (FAST) Navigator, a general framework for action decoding, that achieves state-of-the-art results on the Room-to-Room (R2R) Vision-and-Language navigation challenge of Anderson et.
Ranked #3 on
Vision-Language Navigation
on Room2Room
Prospection: Interpretable Plans From Language By Predicting the Future
High-level human instructions often correspond to behaviors with multiple implicit steps.
Improving Robot Success Detection using Static Object Data
We collect over 13 hours of egocentric manipulation data for training a model to reason about whether a robot successfully placed unseen objects in or on one another.
HellaSwag: Can a Machine Really Finish Your Sentence?
In this paper, we show that commonsense inference still proves difficult for even state-of-the-art models, by presenting HellaSwag, a new challenge dataset.
Ranked #67 on
Sentence Completion
on HellaSwag
Defending Against Neural Fake News
We find that best current discriminators can classify neural fake news from real, human-written, news with 73% accuracy, assuming access to a moderate level of training data.
Ranked #2 on
Fake News Detection
on Grover-Mega
Shifting the Baseline: Single Modality Performance on Visual Navigation \& QA
We demonstrate the surprising strength of unimodal baselines in multimodal domains, and make concrete recommendations for best practices in future research.
Benchmarking Hierarchical Script Knowledge
Understanding procedural language requires reasoning about both hierarchical and temporal relations between events.
Robust Navigation with Language Pretraining and Stochastic Sampling
Core to the vision-and-language navigation (VLN) challenge is building robust instruction representations and action decoding schemes, which can generalize well to previously unseen instructions and environments.
PIQA: Reasoning about Physical Commonsense in Natural Language
Questions requiring this kind of physical commonsense pose a challenge to today's natural language understanding systems.
Ranked #36 on
Question Answering
on PIQA
Natural Language Understanding
Physical Commonsense Reasoning
+1
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
We present ALFRED (Action Learning From Realistic Environments and Directives), a benchmark for learning a mapping from natural language instructions and egocentric vision to sequences of actions for household tasks.
Multi-View Learning for Vision-and-Language Navigation
Learning to navigate in a visual environment following natural language instructions is a challenging task because natural language instructions are highly variable, ambiguous, and under-specified.
Experience Grounds Language
Language understanding research is held back by a failure to relate language to the physical world it describes and to the social interactions it facilitates.
RMM: A Recursive Mental Model for Dialog Navigation
In this paper, we go beyond instruction following and introduce a two-agent task where one agent navigates and asks questions that a second, guiding agent answers.
A Benchmark for Structured Procedural Knowledge Extraction from Cooking Videos
Watching instructional videos are often used to learn about procedures.
The Return of Lexical Dependencies: Neural Lexicalized PCFGs
In this paper we demonstrate that $\textit{context free grammar (CFG) based methods for grammar induction benefit from modeling lexical dependencies}$.
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
ALFWorld enables the creation of a new BUTLER agent whose abstract knowledge, learned in TextWorld, corresponds directly to concrete, visually grounded actions.
RMM: A Recursive Mental Model for Dialogue Navigation
In this paper, we go beyond instruction following and introduce a two-agent task where one agent navigates and asks questions that a second, guiding agent answers.
Imagining Grounded Conceptual Representations from Perceptual Information in Situated Guessing Games
However, as shown by Suglia et al. (2020), existing models fail to learn truly multi-modal representations, relying instead on gold category labels for objects in the scene both at training and inference time.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
Guided by a set of hypotheses, the framework studies how to transform various pre-existing knowledge resources into a form that is most effective for pre-training models.
BUTLER: Building Understanding in TextWorld via Language for Embodied Reasoning
ALFWorld enables the creation of a new BUTLER agent whose abstract knowledge, learned in TextWorld, corresponds directly to concrete, visually grounded actions.
Token-Level Contrast for Video and Language Alignment
Building video and language understanding models requires grounding linguistic concepts and video contents into a shared space.
An Empirical Study on the Generalization Power of Neural Representations Learned via Visual Guessing Games
Guessing games are a prototypical instance of the "learning by interacting" paradigm.
Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Models
Numerous works have analyzed biases in vision and pre-trained language models individually - however, less attention has been paid to how these biases interact in multimodal settings.
Grounding 'Grounding' in NLP
And finally, (3) How to advance our current definition to bridge the gap with Cognitive Science?
Few-shot Language Coordination by Modeling Theory of Mind
Positive results from our experiments hint at the importance of explicitly modeling communication as a socio-pragmatic progress.
Language Grounding with 3D Objects
We introduce several CLIP-based models for distinguishing objects and demonstrate that while recent advances in jointly modeling vision and language are useful for robotic language understanding, it is still the case that these image-based models are weaker at understanding the 3D nature of objects -- properties which play a key role in manipulation.
TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment
This is motivated by the observation that for a video-text pair, the content words in the text, such as nouns and verbs, are more likely to be aligned with the visual contents in the video than the function words.
Ranked #3 on
Temporal Action Localization
on CrossTask
(using extra training data)
WebQA: Multihop and Multimodal QA
Scaling Visual Question Answering (VQA) to the open-domain and multi-hop nature of web searches, requires fundamental advances in visual representation learning, knowledge aggregation, and language generation.
Dependency Induction Through the Lens of Visual Perception
Our experiments find that concreteness is a strong indicator for learning dependency grammars, improving the direct attachment score (DAS) by over 50\% as compared to state-of-the-art models trained on pure text.
Learning When and What to Ask: a Hierarchical Reinforcement Learning Framework
Results on a simulated human-assisted navigation problem demonstrate the effectiveness of our framework: aided with an interaction policy learned by our method, a navigation policy achieves up to a 7× improvement in task success rate compared to performing tasks only by itself.
Hierarchical Reinforcement Learning
reinforcement-learning
+1
Shaped Rewards Bias Emergent Language
The second is shaped rewards which are designed specifically to make the task easier to learn by introducing biases in the learning process.
Symmetric Machine Theory of Mind
Theory of mind (ToM), the ability to understand others' thoughts and desires, is a cornerstone of human intelligence.
FILM: Following Instructions in Language with Modular Methods
In contrast, we propose a modular method with structured representations that (1) builds a semantic map of the scene and (2) performs exploration with a semantic search policy, to achieve the natural language goal.
A Framework for Learning to Request Rich and Contextually Useful Information from Humans
We show that the agent can take advantage of different types of information depending on the context, and analyze the benefits and challenges of learning the assistance-requesting policy when the assistant can recursively decompose tasks into subtasks.
KAT: A Knowledge Augmented Transformer for Vision-and-Language
The primary focus of recent work with largescale transformers has been on optimizing the amount of information packed into the model's parameters.
HEAR: Holistic Evaluation of Audio Representations
The aim of the HEAR benchmark is to develop a general-purpose audio representation that provides a strong basis for learning in a wide variety of tasks and scenarios.
Training Vision-Language Transformers from Captions
Vision-Language Transformers can be learned without low-level human labels (e. g. class labels, bounding boxes, etc).
On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization
Combining the visual modality with pretrained language models has been surprisingly effective for simple descriptive tasks such as image captioning.
Transformers are Adaptable Task Planners
Every home is different, and every person likes things done in their particular way.
Don't Copy the Teacher: Data and Model Challenges in Embodied Dialogue
We provide empirical comparisons of metrics, analysis of three models, and make suggestions for how the field might best progress.
Retrospectives on the Embodied AI Workshop
We present a retrospective on the state of Embodied AI research.
EvEntS ReaLM: Event Reasoning of Entity States via Language Models
This paper investigates models of event implications.
Self-Supervised Object Goal Navigation with In-Situ Finetuning
In contrast, our LocCon shows the most robust transfer in the real world among the set of models we compare to, and that the real-world performance of all models can be further improved with self-supervised LocCon in-situ training.
EXCALIBUR: Encouraging and Evaluating Embodied Exploration
Humans constantly explore and learn about their environment out of curiosity, gather information, and update their models of the world.
The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deployment
In this work, we examine this phenomenon through a series of case studies analyzing the effects of model design decisions, framework paradigms, and hardware platforms on total model latency.
Computational Language Acquisition with Theory of Mind
We also find some evidence that increasing task difficulty in the training process results in more fluent and precise utterances in evaluation.
Spatial-Language Attention Policies for Efficient Robot Learning
We see a 4x improvement over baseline in mobile manipulation setting.
Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agents
We propose the Plan, Eliminate, and Track (PET) framework.
SPRING: Studying the Paper and Reasoning to Play Games
Finally, we show the potential of games as a test bed for LLMs.
HomeRobot: Open-Vocabulary Mobile Manipulation
HomeRobot (noun): An affordable compliant robot that navigates homes and manipulates a wide range of objects in order to complete everyday tasks.
SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMs
In this work, we introduce Semantic Pyramid AutoEncoder (SPAE) for enabling frozen LLMs to perform both understanding and generation tasks involving non-linguistic modalities such as images or videos.
WebArena: A Realistic Web Environment for Building Autonomous Agents
Building upon our environment, we release a set of benchmark tasks focusing on evaluating the functional correctness of task completions.
MAEA: Multimodal Attribution for Embodied AI
Understanding multimodal perception for embodied AI is an open question because such inputs may contain highly complementary as well as redundant information for the task.
Reasoning about the Unseen for Efficient Outdoor Object Navigation
We introduce a new task OUTDOOR, a new mechanism for Large Language Models (LLMs) to accurately hallucinate possible futures, and a new computationally aware success metric for pushing research forward in this more complex domain.
SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents
We present SOTOPIA, an open-ended environment to simulate complex social interactions between artificial agents and evaluate their social intelligence.
Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis
Motivated by the impressive open-set performance and content generation capabilities of web-scale, large-capacity pre-trained models (i. e., foundation models) in research fields such as Natural Language Processing (NLP) and Computer Vision (CV), we devote this survey to exploring (i) how these existing foundation models from NLP and CV can be applied to the field of robotics, and also exploring (ii) what a robotics-specific foundation model would look like.
VISREAS: Complex Visual Reasoning with Unanswerable Questions
The unique feature of this task, validating question answerability with respect to an image before answering, and the poor performance of state-of-the-art models inspired the design of a new modular baseline, LOGIC2VISION that reasons by producing and executing pseudocode without any external modules to generate the answer.
SOTOPIA-$π$: Interactive Learning of Socially Intelligent Language Agents
Motivated by this gap, we propose an interactive learning method, SOTOPIA-$\pi$, improving the social intelligence of language agents.
Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers
Given a video demonstration of a manipulation task and current visual observations, Vid2Robot directly produces robot actions.
Skews in the Phenomenon Space Hinder Generalization in Text-to-Image Generation
We hypothesize that the underlying phenomenological coverage has not been proportionally scaled up, leading to a skew of the presented phenomenon which harms generalization.
Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward
Preference modeling techniques, such as direct preference optimization (DPO), has shown effective in enhancing the generalization abilities of large language model (LLM).
Dialogue with Robots: Proposals for Broadening Participation and Research in the SLIVAR Community
The ability to interact with machines using natural human language is becoming not just commonplace, but expected.
AgentKit: Flow Engineering with Graphs, not Coding
The chains of nodes can be designed to explicitly enforce a naturally structured "thought process".
