Search Results for author:
Found 83 papers, 43 papers with code
Guided K-best Selection for Semantic Parsing Annotation
Collecting data for conversational semantic parsing is a time-consuming and demanding process.
How Do Neural Sequence Models Generalize? Local and Global Cues for Out-of-Distribution Prediction
After a neural sequence model encounters an unexpected token, can its behavior be predicted?
Decision-Oriented Dialogue for Human-AI Collaboration
In each of these settings, AI assistants and users have disparate abilities that they must combine to arrive at the best decision: assistants can access and process large amounts of information, while users have preferences and constraints external to the system.
Grokking of Hierarchical Structure in Vanilla Transformers
When analyzing the relationship between model-internal properties and grokking, we find that optimal depth for grokking can be identified using the tree-structuredness metric of \citet{murty2023projections}.
Natural Language Decomposition and Interpretation of Complex Utterances
We introduce an approach for equipping a simple language-to-code model to handle complex utterances via a process of hierarchical natural language decomposition.
Inspecting and Editing Knowledge Representations in Language Models
Neural language models (LMs) represent facts about the world described by text.
Language Models Trained on Media Diets Can Predict Public Opinion
Public opinion reflects and shapes societal behavior, but the traditional survey-based tools to measure it are limited.
Guiding Pretraining in Reinforcement Learning with Large Language Models
Reinforcement learning algorithms typically struggle in the absence of a dense, well-shaped reward function.
LaMPP: Language Models as Probabilistic Priors for Perception and Action
Language models trained on large text corpora encode rich distributional information about real-world environments and action sequences.
Language Modeling with Latent Situations
Language models (LMs) often generate incoherent outputs: they refer to events and entity states that are incompatible with the state of the world described in their inputs.
PromptBoosting: Black-Box Text Classification with Ten Forward Passes
Instead of directly optimizing in prompt space, PromptBoosting obtains a small pool of prompts via a gradient-free approach and then constructs a large pool of weak learners by pairing these prompts with different elements of the LM's output distribution.
Language Models as Agent Models
Language models (LMs) are trained on collections of documents, written by individual human agents to achieve specific goals in an outside world.
What learning algorithm is in-context learning? Investigations with linear models
We investigate the hypothesis that transformer-based in-context learners implement standard learning algorithms implicitly, by encoding smaller models in their activations, and updating these implicit models as new examples appear in the context.
Hierarchical Phrase-based Sequence-to-Sequence Learning
We describe a neural transducer that maintains the flexibility of standard sequence-to-sequence (seq2seq) models while incorporating hierarchical phrases as a source of inductive bias during training and as explicit constraints during inference.
Characterizing Intrinsic Compositionality in Transformers with Tree Projections
To evaluate this possibility, we describe an unsupervised and parameter-free method to \emph{functionally project} the behavior of any transformer into the space of tree-structured networks.
ObSynth: An Interactive Synthesis System for Generating Object Models from Natural Language Specifications
We introduce ObSynth, an interactive system leveraging the domain knowledge embedded in large language models (LLMs) to help users design object models from high level natural language prompts.
The Whole Truth and Nothing But the Truth: Faithful and Controllable Dialogue Response Generation with Dataflow Transduction and Constrained Decoding
Satisfying these constraints simultaneously is difficult for the two predominant paradigms in language generation: neural language modeling and rule-based generation.
Towards Tracing Factual Knowledge in Language Models Back to the Training Data
In this paper, we propose the problem of fact tracing: identifying which training examples taught an LM to generate a particular factual assertion.
Identifying concept libraries from language about object structure
Our understanding of the visual world goes beyond naming objects, encompassing our ability to parse objects into meaningful parts, attributes, and relations.
Correcting Robot Plans with Natural Language Feedback
In this paper, we explore natural language as an expressive and flexible tool for robot correction.
Teachable Reinforcement Learning via Advice Distillation
Training automated agents to complete complex tasks in interactive environments is challenging: reinforcement learning requires careful hand-engineering of reward functions, imitation learning requires specialized infrastructure and access to a human expert, and learning from intermediate forms of supervision (like binary preferences) is time-consuming and extracts little information from each human intervention.
Pre-Trained Language Models for Interactive Decision-Making
Together, these results suggest that language modeling induces representations that are useful for modeling not just language, but also goals and plans; these representations can aid learning and generalization even outside of language processing.
Compositionality as Lexical Symmetry
Standard deep network models lack the inductive biases needed to generalize compositionally in tasks like semantic parsing, translation, and question answering.
Natural Language Descriptions of Deep Visual Features
Given a neuron, MILAN generates a description by searching for a natural language string that maximizes pointwise mutual information with the image regions in which the neuron is active.
Modeling Strong and Human-Like Gameplay with KL-Regularized Search
We consider the task of building strong but human-like policies in multi-agent decision-making problems, given examples of human behavior.
Quantifying Adaptability in Pre-trained Language Models with 500 Tasks
When a neural language model (LM) is adapted to perform a new task, what aspects of the task predict the eventual performance of the model?
How Do Neural Sequence Models Generalize? Local and Global Context Cues for Out-of-Distribution Prediction
After a neural sequence model encounters an unexpected token, can its behavior be predicted?
Subspace Regularizers for Few-Shot Class Incremental Learning
The key to this approach is a new family of subspace regularization schemes that encourage weight vectors for new classes to lie close to the subspace spanned by the weights of existing classes.
class-incremental learning
Few-Shot Class-Incremental Learning
+3
Toward a Visual Concept Vocabulary for GAN Latent Space
A large body of recent work has identified transformations in the latent spaces of generative adversarial networks (GANs) that consistently and interpretably transform generated images.
Skill Induction and Planning with Latent Language
We evaluate this approach in the ALFRED household simulation environment, providing natural language annotations for only 10% of demonstrations.
Natural Language Descriptions of Deep Features
Given a neuron, MILAN generates a description by searching for a natural language string that maximizes pointwise mutual information with the image regions in which the neuron is active.
Language Model Pre-training Improves Generalization in Policy Learning
Additional experiments explore the role of language-based encodings in these results; we find that it is possible to train a simple adapter layer that maps from observations and action histories to LM embeddings, and thus that language modeling provides an effective initializer even for tasks with no language as input or output.
Lexicon Learning for Few Shot Sequence Modeling
Sequence-to-sequence transduction is the core problem in language processing applications as diverse as semantic parsing, machine translation, and instruction following.
Value-Agnostic Conversational Semantic Parsing
Conversational semantic parsers map user utterances to executable programs given dialogue histories composed of previous utterances, programs, and system responses.
FairyTailor: A Multimodal Generative Framework for Storytelling
Natural language generation (NLG) for storytelling is especially challenging because it requires the generated text to follow an overall theme while remaining creative and diverse to engage the reader.
Leveraging Language to Learn Program Abstractions and Search Heuristics
Inductive program synthesis, or inferring programs from examples of desired behavior, offers a general paradigm for building interpretable, robust, and generalizable machine learning systems.
What Context Features Can Transformer Language Models Use?
Transformer-based language models benefit from conditioning on contexts of hundreds to thousands of previous tokens.
Lexicon Learning for Few-Shot Neural Sequence Modeling
Sequence-to-sequence transduction is the core problem in language processing applications as diverse as semantic parsing, machine translation, and instruction following.
Compositional Generalization for Neural Semantic Parsing via Span-level Supervised Attention
We describe a span-level supervised attention loss that improves compositional generalization in semantic parsers.
Implicit Representations of Meaning in Neural Language Models
Does the effectiveness of neural language models derive entirely from accurate modeling of surface word co-occurrence statistics, or do these models represent and reason about the world they describe?
The Low-Dimensional Linear Geometry of Contextualized Word Representations
We show that a variety of linguistic features (including structured dependency relationships) are encoded in low-dimensional subspaces.
Cetacean Translation Initiative: a roadmap to deciphering the communication of sperm whales
We posit that machine learning will be the cornerstone of future collection, processing, and analysis of multimodal streams of data in animal communication studies, including bioacoustic, behavioral, biological, and environmental data.
Multitasking Inhibits Semantic Drift
When intelligent agents communicate to accomplish shared goals, how do these goals shape the agents' language?
Representing Partial Programs with Blended Abstract Semantics
In this search process, a key challenge is representing the behavior of a partially written program before it can be executed, to judge if it is on the right track and predict where to search next.
Learning to Recombine and Resample Data for Compositional Generalization
Flexible neural sequence models outperform grammar- and automaton-based counterparts on a variety of tasks.
Task-Oriented Dialogue as Dataflow Synthesis
We describe an approach to task-oriented dialogue in which dialogue state is represented as a dataflow graph.
Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessment
To take advantage of the rich information present in the radiology reports, we develop a neural network model that is trained on both images and free-text to assess pulmonary edema severity from chest radiographs at inference time.
Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Prediction
We present a randomized controlled trial for a model-in-the-loop regression task, with the goal of measuring the extent to which (1) good explanations of model predictions increase human accuracy, and (2) faulty explanations decrease human trust in the model.
Compositional Explanations of Neurons
We describe a procedure for explaining neurons in deep representations by identifying compositional logical concepts that closely approximate neuron behavior.
Unnatural Language Processing: Bridging the Gap Between Synthetic and Natural Language Data
Large, human-annotated datasets are central to the development of natural language processing models.
Experience Grounds Language
Language understanding research is held back by a failure to relate language to the physical world it describes and to the social interactions it facilitates.
A Benchmark for Systematic Generalization in Grounded Language Understanding
In this paper, we introduce a new benchmark, gSCAN, for evaluating compositional generalization in situated language understanding.
A Survey of Reinforcement Learning Informed by Natural Language
To be successful in real-world tasks, Reinforcement Learning (RL) needs to exploit the compositional, relational, and hierarchical structure of the world, and learn to transfer it to the task at hand.
Good-Enough Compositional Data Augmentation
We propose a simple data augmentation protocol aimed at providing a compositional inductive bias in conditional and unconditional sequence models.
Pragmatically Informative Text Generation
We improve the informativeness of models for conditional text generation using techniques from computational pragmatics.
Ranked #1 on
Data-to-Text Generation
on E2E NLG Challenge
Abstractive Text Summarization
Conditional Text Generation
+3
Measuring Compositionality in Representation Learning
Many machine learning algorithms represent input data with vector embeddings or discrete codes.
Guiding Policies with Language via Meta-Learning
However, a single instruction may be insufficient to fully communicate our intent or, even if it is, may be insufficient for an autonomous agent to actually understand how to perform the desired task.
Explainable Neural Computation via Stack Neural Module Networks
In complex inferential tasks like question answering, machine learning models must confront two challenges: the need to implement a compositional reasoning process, and, in many applications, the need for this reasoning process to be interpretable to assist users in both development and prediction.
Ranked #13 on
Referring Expression Comprehension
on Talk2Car
Speaker-Follower Models for Vision-and-Language Navigation
We use this speaker model to (1) synthesize new instructions for data augmentation and to (2) implement pragmatic reasoning, which evaluates how well candidate action sequences explain an instruction.
Unified Pragmatic Models for Generating and Following Instructions
We show that explicit pragmatic inference aids in correctly generating and following natural language instructions for complex, sequential tasks.
Can Deep Reinforcement Learning Solve Erdos-Selfridge-Spencer Games?
Deep reinforcement learning has achieved many recent successes, but our understanding of its strengths and limitations is hampered by the lack of rich environments in which we can fully characterize optimal behavior, and correspondingly diagnose individual actions against such a characterization.
Learning with Latent Language
The named concepts and compositional operators present in natural language provide a rich source of information about the kinds of abstractions humans use to navigate the world.
Analogs of Linguistic Structure in Deep Representations
We investigate the compositional structure of message vectors computed by a deep network trained on a communication game.
A Minimal Span-Based Neural Constituency Parser
In this work, we present a minimal neural model for constituency parsing based on independent scoring of labels and spans.
Translating Neuralese
Several approaches have recently been proposed for learning decentralized deep multiagent policies that coordinate via a differentiable communication channel.
Learning to Reason: End-to-End Module Networks for Visual Question Answering
Natural language questions are inherently compositional, and many are most easily answered by reasoning about their decomposition into modular sub-problems.
Ranked #45 on
Visual Question Answering (VQA)
on VQA v2 test-dev
Modeling Relationships in Referential Expressions with Compositional Modular Networks
In this paper we instead present a modular deep architecture capable of analyzing referential expressions into their component parts, identifying entities and relationships mentioned in the input expression and grounding them all in the scene.
Modular Multitask Reinforcement Learning with Policy Sketches
We describe a framework for multitask deep reinforcement learning guided by policy sketches.
Reasoning About Pragmatics with Neural Listeners and Speakers
We present a model for pragmatically describing scenes, in which contrastive behavior results from a combination of inference-driven pragmatics and learned semantics.
Learning to Compose Neural Networks for Question Answering
We describe a question answering model that applies to both images and structured knowledge bases.
Neural Module Networks
Visual question answering is fundamentally compositional in nature---a question like "where is the dog?"
Ranked #6 on
Visual Question Answering (VQA)
on VQA v1 test-std
Alignment-based compositional semantics for instruction following
This paper describes an alignment-based model for interpreting natural language instructions in context.
On the accuracy of self-normalized log-linear models
Calculation of the log-normalizer is a major computational obstacle in applications of log-linear models with large output spaces.
Unsupervised Transcription of Piano Music
We present a new probabilistic model for transcribing piano music from audio to a symbolic form.
Annotating Agreement and Disagreement in Threaded Discussion
We introduce a new corpus of sentence-level agreement and disagreement annotations over LiveJournal and Wikipedia threads.
