Search Results for author:
Found 32 papers, 20 papers with code
Crowdsourcing Beyond Annotation: Case Studies in Benchmark Data Collection
Even though it is such a fundamental tool in NLP, crowdsourcing use is largely guided by common practices and the personal experience of researchers.
Learning Adaptive Parallel Reasoning with Language Models
Scaling inference-time computation has substantially improved the reasoning capabilities of language models.
TULIP: Towards Unified Language-Image Pretraining
These models, by performing language alignment, tend to prioritize high-level semantics over visual understanding, weakening their image understanding.
AutoPresent: Designing Structured Visuals from Scratch
We benchmark end-to-end image generation and program generation methods with a variety of models, and find that programmatic methods produce higher-quality slides in user-interactable formats.
Training Software Engineering Agents and Verifiers with SWE-Gym
When combined with our fine-tuned SWE agents, we achieve 32. 0% and 26. 0% on SWE-Bench Verified and Lite, respectively, reflecting a new state-of-the-art for open-weight SWE agents.
Evaluating Model Perception of Color Illusions in Photorealistic Scenes
We study the perception of color illusions by vision-language models.
Using Language Models to Disambiguate Lexical Choices in Translation
In translation, a concept represented by a single word in a source language can have multiple variations in a target language.
Grounding Language in Multi-Perspective Referential Communication
We introduce a task and dataset for referring expression generation and comprehension in multi-agent embodied environments.
DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning
This paper introduces a novel autonomous RL approach, called DigiRL, for training in-the-wild device control agents through fine-tuning a pre-trained VLM in two stages: offline RL to initialize the model, followed by offline-to-online RL.
Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning
Finally, our framework uses these task rewards to fine-tune the entire VLM with RL.
Autonomous Evaluation and Refinement of Digital Agents
We show that domain-general automatic evaluators can significantly improve the performance of agents for web navigation and device control.
UNcommonsense Reasoning: Abductive Reasoning about Uncommon Situations
To instead investigate the ability to model unusual, unexpected, and unlikely situations, we explore the task of uncommonsense abductive reasoning.
What's In My Big Data?
We open-source WIMBD's code and artifacts to provide a standard set of evaluations for new text-based corpora and to encourage more analyses and transparency around them.
Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting
In this work, we focus on LLM sensitivity to a quintessential class of meaning-preserving design choices: prompt formatting.
Fine-Grained Human Feedback Gives Better Rewards for Language Model Training
We introduce Fine-Grained RLHF, a framework that enables training and learning from reward functions that are fine-grained in two respects: (1) density, providing a reward after every segment (e. g., a sentence) is generated; and (2) incorporating multiple reward models associated with different feedback types (e. g., factual incorrectness, irrelevance, and information incompleteness).
Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker
We present SymbolicToM, a plug-and-play approach to reason about the belief states of multiple characters in reading comprehension tasks via explicit symbolic representation.
We're Afraid Language Models Aren't Modeling Ambiguity
We find that the task remains extremely challenging, including for GPT-4, whose generated disambiguations are considered correct only 32% of the time in human evaluation, compared to 90% for disambiguations in our dataset.
Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modelling
Reinforcement learning (RL) agents typically learn tabula rasa, without prior knowledge of the world.
Continual Learning for Instruction Following from Realtime Feedback
We propose and deploy an approach to continually train an instruction-following agent from feedback provided by users during collaborative interactions.
Abstract Visual Reasoning with Tangram Shapes
We introduce KiloGram, a resource for studying abstract visual reasoning in humans and machines.
Analysis of Language Change in Collaborative Instruction Following
We analyze language change over time in a collaborative, goal-oriented instructional task, where utility-maximizing participants form conventions and increase their expertise.
Continual Learning for Grounded Instruction Generation by Observing Human Following Behavior
We study continual learning for natural language instruction generation, by observing human users' instruction execution.
Exploring Unexplored Generalization Challenges for Cross-Database Semantic Parsing
We study the task of cross-database semantic parsing (XSP), where a system that maps natural language utterances to executable SQL queries is evaluated on databases unseen during training.
Executing Instructions in Situated Collaborative Interactions
We study a collaborative scenario where a user not only instructs a system to complete tasks, but also acts alongside it.
NLVR2 Visual Bias Analysis
We show that the performance of existing models (Li et al., 2019; Tan and Bansal 2019) is relatively robust to this potential bias.
Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments
We study the problem of jointly reasoning about language and vision through a navigation and spatial reasoning task.
Ranked #11 on
Vision and Language Navigation
on Touchdown Dataset
A Corpus for Reasoning About Natural Language Grounded in Photographs
We crowdsource the data using sets of visually rich images and a compare-and-contrast task to elicit linguistically diverse language.
Neural Semantic Parsing
Semantic parsing, the study of translating natural language utterances into machine-executable programs, is a well-established research area and has applications in question answering, instruction following, voice assistants, and code generation.
Situated Mapping of Sequential Instructions to Actions with Single-step Reward Observation
We propose a learning approach for mapping context-dependent sequential instructions to actions.
Learning to Map Context-Dependent Sentences to Executable Formal Queries
We propose a context-dependent model to map utterances within an interaction to executable formal queries.
Visual Reasoning with Natural Language
To understand language in complex environments, agents must reason about the full range of language inputs and their correspondence to the world.
A Corpus of Natural Language for Visual Reasoning
We present a new visual reasoning language dataset, containing 92, 244 pairs of examples of natural statements grounded in synthetic images with 3, 962 unique sentences.
