Search Results for author:
Found 217 papers, 157 papers with code
Learning Semantic Correspondences with Less Supervision
A central problem in grounded language acquisition is learning the correspondences between a rich world state and a stream of text which references that world state.
Spectral Experts for Estimating Mixtures of Linear Regressions
Discriminative latent-variable models are typically learned using EM or gradient-based optimization, which suffer from local optima.
Dropout Training as Adaptive Regularization
Dropout and other feature noising schemes control overfitting by artificially corrupting the training data.
Lambda Dependency-Based Compositional Semantics
This short note presents a new formal language, lambda dependency-based compositional semantics (lambda DCS) for representing logical forms in semantic parsing.
Relaxations for inference in restricted Boltzmann machines
We propose a relaxation-based approximate inference algorithm that samples near-MAP configurations of a binary pairwise Markov random field.
Altitude Training: Strong Bounds for Single-Layer Dropout
Dropout training, originally designed for deep neural networks, has been successful on high-dimensional single-layer natural language tasks.
The Statistics of Streaming Sparse Regression
We present a sparse analogue to stochastic gradient descent that is guaranteed to perform well under similar conditions to the lasso.
Imitation Learning of Agenda-based Semantic Parsers
Semantic parsers conventionally construct logical forms bottom-up in a fixed order, resulting in the generation of many extraneous partial logical forms.
Tensor Factorization via Matrix Factorization
Tensor factorization arises in many machine learning applications, such knowledge base modeling and parameter estimation in latent variable models.
Learning Fast-Mixing Models for Structured Prediction
Markov Chain Monte Carlo (MCMC) algorithms are often used for approximate inference inside learning, but their slow mixing can be difficult to diagnose and the approximations can seriously degrade learning.
Reified Context Models
A classic tension exists between exact inference in a simple model and approximate inference in a complex model.
Learning Where to Sample in Structured Prediction
In structured prediction, most inference algorithms allocate a homogeneous amount of computation to all parts of the output, which can be wasteful when different parts vary widely in terms of difficulty.
Traversing Knowledge Graphs in Vector Space
Path queries on a knowledge graph can be used to answer compositional questions such as "What languages are spoken by people living in Lisbon?".
On-the-Job Learning with Bayesian Decision Theory
Our goal is to deploy a high-accuracy system starting with zero training examples.
Compositional Semantic Parsing on Semi-Structured Tables
Two important aspects of semantic parsing for question answering are the breadth of the knowledge source and the depth of logical compositionality.
Data Augmentation via Levy Processes
The framework imagines data as being drawn from a slice of a Levy process.
Learning Executable Semantic Parsers for Natural Language Understanding
For building question answering systems and natural language interfaces, semantic parsing has emerged as an important and powerful paradigm.
Estimating Mixture Models via Mixtures of Polynomials
This framework allows us to draw insights and apply tools from convex optimization, computer algebra and the theory of moments to study problems in statistical estimation.
Learning Language Games through Interaction
We introduce a new language learning setting relevant to building adaptive natural language interfaces.
Data Recombination for Neural Semantic Parsing
Modeling crisp logical regularities is crucial in semantic parsing, making it difficult for neural models with no task-specific prior knowledge to achieve good results.
SQuAD: 100,000+ Questions for Machine Comprehension of Text
We present the Stanford Question Answering Dataset (SQuAD), a new reading comprehension dataset consisting of 100, 000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage.
Simpler Context-Dependent Logical Forms via Model Projections
With only denotations at training time, we must search over a combinatorially large space of logical forms, which is even larger with context-dependent utterances.
Unsupervised Risk Estimation Using Only Conditional Independence Structure
We show how to estimate a model's test error from unlabeled data, on distributions very different from the training distribution, while assuming only that certain conditional independencies are preserved between train and test.
Unanimous Prediction for 100% Precision with Application to Learning Semantic Mappings
Specifically, we introduce the unanimity principle: only predict when all models consistent with the training data predict the same output.
Inferring Logical Forms From Denotations
A core problem in learning semantic parsers from denotations is picking out consistent logical forms--those that yield the correct denotation--from a combinatorially large space.
Synthesizing Program Input Grammars
We present an algorithm for synthesizing a context-free grammar encoding the language of valid program inputs from a set of input examples and blackbox access to the program.
Programming Languages
Estimation from Indirect Supervision with Linear Moments
In structured prediction problems where we have indirect supervision of the output, maximum marginal likelihood faces two computational obstacles: non-convexity of the objective and intractability of even a single gradient computation.
How Much is 131 Million Dollars? Putting Numbers in Perspective with Compositional Descriptions
We then propose a system to generate these descriptions consisting of two steps: formula construction and description generation.
Convexified Convolutional Neural Networks
For learning two-layer convolutional neural networks, we prove that the generalization error obtained by a convexified CNN converges to that of the best possible CNN.
Prediction with a Short Memory
For a Hidden Markov Model with $n$ hidden states, $I$ is bounded by $\log n$, a quantity that does not depend on the mixing time, and we show that the trivial prediction algorithm based on the empirical frequencies of length $O(\log n/\epsilon)$ windows of observations achieves this error, provided the length of the sequence is $d^{\Omega(\log n/\epsilon)}$, where $d$ is the size of the observation alphabet.
A Hitting Time Analysis of Stochastic Gradient Langevin Dynamics
We study the Stochastic Gradient Langevin Dynamics (SGLD) algorithm for non-convex optimization.
Understanding Black-box Predictions via Influence Functions
How can we explain the predictions of a black-box model?
Naturalizing a Programming Language via Interactive Learning
Our goal is to create a convenient natural language interface for performing well-specified but complex actions such as analyzing data, manipulating text, and querying databases.
Learning Symmetric Collaborative Dialogue Agents with Dynamic Knowledge Graph Embeddings
To model both structured knowledge and unstructured language, we propose a neural model with dynamic knowledge graph embeddings that evolve as the dialogue progresses.
From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihood
Our goal is to learn a semantic parser that maps natural language utterances into executable programs when only indirect supervision is available: examples are labeled with the correct execution result, but not the program itself.
Certified Defenses for Data Poisoning Attacks
Machine learning systems trained on user-provided data are susceptible to data poisoning attacks, whereby malicious users inject false training data with the aim of corrupting the learned model.
Developing Bug-Free Machine Learning Systems With Formal Mathematics
As a case study, we implement a new system, Certigrad, for optimizing over stochastic computation graphs, and we generate a formal (i. e. machine-checkable) proof that the gradients sampled by the system are unbiased estimates of the true mathematical gradients.
Adversarial Examples for Evaluating Reading Comprehension Systems
Standard accuracy metrics indicate that reading comprehension systems are making rapid progress, but the extent to which these systems truly understand language remains unclear.
Macro Grammars and Holistic Triggering for Efficient Semantic Parsing
To learn a semantic parser from denotations, a learning algorithm must search over a combinatorially large space of logical forms for ones consistent with the annotated denotations.
World of Bits: An Open-Domain Platform for Web-Based Agents
While simulated game environments have greatly accelerated research in reinforcement learning, existing environments lack the open-domain realism of tasks in computer vision or natural language processing, which operate on artifacts created by humans in natural, organic settings.
Importance sampling for unbiased on-demand evaluation of knowledge base population
Our first contribution is a new importance-sampling based evaluation which corrects for this bias by annotating a new system{'}s predictions on-demand via crowdsourcing.
Generating Sentences by Editing Prototypes
We propose a new generative model of sentences that first samples a prototype sentence from the training corpus and then edits it into a new sentence.
Unsupervised Transformation Learning via Convex Relaxations
Our goal is to extract meaningful transformations from raw images, such as varying the thickness of lines in handwriting or the lighting in a portrait.
Learning Overcomplete HMMs
On the other hand, we show that learning is impossible given only a polynomial number of samples for HMMs with a small output alphabet and whose transition matrices are random regular graphs with large degree.
Certified Defenses against Adversarial Examples
While neural networks have achieved high accuracy on standard image classification benchmarks, their accuracy drops to nearly zero in the presence of small adversarial perturbations to test inputs.
Learning a SAT Solver from Single-Bit Supervision
We present NeuroSAT, a message passing neural network that learns to solve SAT problems after only being trained as a classifier to predict satisfiability.
Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration
Reinforcement learning (RL) agents improve through trial-and-error, but when reward is sparse and the agent cannot discover successful action sequences, learning stagnates.
Generalized Binary Search For Split-Neighborly Problems
In sequential hypothesis testing, Generalized Binary Search (GBS) greedily chooses the test with the highest information gain at each step.
Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfer
We consider the task of text attribute transfer: transforming a sentence to alter a specific attribute (e. g., sentiment) while preserving its attribute-independent content (e. g., changing "screen is just the right size" to "screen is too small").
Ranked #1 on
Unsupervised Text Style Transfer
on Yelp2018
Training Classifiers with Natural Language Explanations
Training accurate classifiers requires many labels, but each label provides only limited information (one bit for binary classification).
Planning, Inference and Pragmatics in Sequential Language Games
We study sequential language games in which two players, each with private information, communicate to achieve a common goal.
Know What You Don't Know: Unanswerable Questions for SQuAD
Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for which the correct answer is not stated in the context.
On the Relationship between Data Efficiency and Error for Uncertainty Sampling
While active learning offers potential cost savings, the actual data efficiency---the reduction in amount of labeled data needed to obtain the same error rate---observed in practice is mixed.
Fairness Without Demographics in Repeated Loss Minimization
Machine learning models (e. g., speech recognizers) are usually trained to minimize average loss, which results in representation disparity---minority groups (e. g., non-native speakers) contribute less to the training objective and thus tend to suffer higher loss.
The price of debiasing automatic metrics in natural language evalaution
For evaluating generation systems, automatic metrics such as BLEU cost nothing to run but have been shown to correlate poorly with human judgment, leading to systematic bias against certain model improvements.
The price of debiasing automatic metrics in natural language evaluation
For evaluating generation systems, automatic metrics such as BLEU cost nothing to run but have been shown to correlate poorly with human judgment, leading to systematic bias against certain model improvements.
Inferring Multidimensional Rates of Aging from Cross-Sectional Data
Motivated by the study of human aging, we present an interpretable latent-variable model that learns temporal dynamics from cross-sectional data.
QuAC : Question Answering in Context
We present QuAC, a dataset for Question Answering in Context that contains 14K information-seeking QA dialogs (100K questions in total).
Mapping Natural Language Commands to Web Elements
The web provides a rich, open-domain environment with textual, structural, and spatial properties.
Decoupling Strategy and Generation in Negotiation Dialogues
We consider negotiation settings in which two agents use natural language to bargain on goods.
Textual Analogy Parsing: What's Shared and What's Compared among Analogous Facts
To understand a sentence like "whereas only 10% of White Americans live at or below the poverty line, 28% of African Americans do" it is important not only to identify individual facts, e. g., poverty rates of distinct demographic groups, but also the higher-order relations between them, e. g., the disparity between them.
Transforming Question Answering Datasets Into Natural Language Inference Datasets
Existing datasets for natural language inference (NLI) have propelled research on language understanding.
QuAC: Question Answering in Context
We present QuAC, a dataset for Question Answering in Context that contains 14K information-seeking QA dialogs (100K questions in total).
Semidefinite relaxations for certifying robustness to adversarial examples
One promise of ending the arms race is developing certified defenses, ones which are provably robust against all attackers in some family.
Stronger Data Poisoning Attacks Break Data Sanitization Defenses
In this paper, we develop three attacks that can bypass a broad range of common data sanitization defenses, including anomaly detectors based on nearest neighbors, training loss, and singular-value decomposition.
FrAngel: Component-Based Synthesis with Control Structures
We present FrAngel, a new approach to component-based synthesis that can synthesize short Java functions with control structures when given a desired signature, a set of input-output examples, and a collection of libraries (without formal specifications).
Programming Languages
A Retrieve-and-Edit Framework for Predicting Structured Outputs
For the task of generating complex outputs such as source code, editing existing outputs can be easier than generating complex outputs from scratch.
Uncertainty Sampling is Preconditioned Stochastic Gradient Descent on Zero-One Loss
Uncertainty sampling, a popular active learning algorithm, is used to reduce the amount of data required to learn a classifier, but it has been observed in practice to converge to different parameters depending on the initialization and sometimes to even better parameters than standard training on all the data.
Defending against Whitebox Adversarial Attacks via Randomized Discretization
Adversarial perturbations dramatically decrease the accuracy of state-of-the-art image classifiers.
Unifying Human and Statistical Evaluation for Natural Language Generation
How can we measure whether a natural language generation system produces both high quality and diverse outputs?
Pun Generation with Surprise
We tackle the problem of generating a pun sentence given a pair of homophones (e. g., "died" and "dyed").
Select Via Proxy: Efficient Data Selection For Training Deep Networks
In our approach, we first train a small proxy model quickly, which we then use to estimate the utility of individual training data points, and then select the most informative ones for training the large target model.
Learning Abstract Models for Long-Horizon Exploration
In our approach, a manager maintains an abstract MDP over a subset of the abstract states, which grows monotonically through targeted exploration (possible due to the abstract MDP).
Strategies for Pre-training Graph Neural Networks
Many applications of machine learning require a model to make accurate pre-dictions on test examples that are distributionally different from training ones, while task-specific labels are scarce during training.
Ranked #3 on
Molecular Property Prediction
on ToxCast
On the Accuracy of Influence Functions for Measuring Group Effects
Influence functions estimate the effect of removing a training point on a model without the need to retrain.
Unlabeled Data Improves Adversarial Robustness
We demonstrate, theoretically and empirically, that adversarial robustness can significantly benefit from semisupervised learning.
Maximum Weighted Loss Discrepancy
To capture this inequality, we introduce and study a notion we call maximum weighted loss discrepancy (MWLD), the maximum (weighted) difference between the loss of a group and the loss of the population.
SPoC: Search-based Pseudocode to Code
Given test cases as a mechanism to validate programs, we search over the space of possible translations of the pseudocode to find a program that passes the validation.
Ranked #2 on
Program Synthesis
on SPoC TestP
Adversarial Training Can Hurt Generalization
While adversarial training can improve robust accuracy (against an adversary), it sometimes hurts standard accuracy (when there is no adversary).
A Tight Analysis of Greedy Yields Subexponential Time Approximation for Uniform Decision Tree
The greedy algorithm's $O(\log n)$ approximation ratio was the best known, but the largest approximation ratio known to be NP-hard is $4-\varepsilon$.
Selection via Proxy: Efficient Data Selection for Deep Learning
By removing hidden layers from the target model, using smaller architectures, and training for fewer epochs, we create proxies that are an order of magnitude faster to train.
Certified Robustness to Adversarial Word Substitutions
We train the first models that are provably robust to all word substitutions in this family.
Distributionally Robust Language Modeling
Language models are generally trained on data spanning a wide range of topics (e. g., news, reviews, fiction), but they might be applied to an a priori unknown target distribution (e. g., restaurant reviews).
Designing and Interpreting Probes with Control Tasks
The selectivity of a probe puts linguistic task accuracy in context with the probe's capacity to memorize from word types.
Verified Uncertainty Calibration
In these experiments, we also estimate the calibration error and ECE more accurately than the commonly used plugin estimators.
When Covariate-shifted Data Augmentation Increases Test Error And How to Fix It
Empirically, data augmentation sometimes improves and sometimes hurts test error, even when only adding points with labels from the true conditional distribution that the hypothesis class is expressive enough to fit.
Shaping Visual Representations with Language for Few-shot Classification
By describing the features and abstractions of our world, language is a crucial tool for human learning and a promising source of supervision for machine learning models.
Learning Autocomplete Systems as a Communication Game
We study textual autocomplete---the task of predicting a full sentence from a partial sentence---as a human-machine communication game.
Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
Distributionally robust optimization (DRO) allows us to learn models that instead minimize the worst-case training loss over a set of pre-defined groups.
Ranked #1 on
Out-of-Distribution Generalization
on UrbanCars
Feature Noise Induces Loss Discrepancy Across Groups
Our main result is that even when there is no information deficiency specific to one group (e. g., both groups have infinite data), adding the same amount of feature noise to all individuals leads to loss discrepancy.
Understanding and Mitigating the Tradeoff Between Robustness and Accuracy
In this work, we precisely characterize the effect of augmentation on the standard error in linear regression when the optimal linear predictor has zero standard and robust error.
Understanding Self-Training for Gradual Domain Adaptation
Machine learning systems must adapt to data distributions that evolve over time, in applications ranging from sensor networks and self-driving car perception modules to brain-machine interfaces.
Distributionally Robust Neural Networks
Distributionally robust optimization (DRO) allows us to learn models that instead minimize the worst-case training loss over a set of pre-defined groups.
Robust Encodings: A Framework for Combating Adversarial Typos
We instantiate RobEn to defend against a large family of adversarial typos.
ExpBERT: Representation Engineering with Natural Language Explanations
Suppose we want to specify the inductive bias that married couples typically go on honeymoons for the task of extracting pairs of spouses from text.
An Investigation of Why Overparameterization Exacerbates Spurious Correlations
We study why overparameterization -- increasing model size well beyond the point of zero training error -- can hurt test error on minority groups despite improving average test error when there are spurious correlations in the data.
Enabling Language Models to Fill in the Blanks
We show that this approach, which we call infilling by language modeling, can enable LMs to infill entire sentences effectively on three different domains: short stories, scientific abstracts, and lyrics.
Graph-based, Self-Supervised Program Repair from Diagnostic Feedback
Second, we present a self-supervised learning paradigm for program repair that leverages unlabeled programs available online to create a large amount of extra program repair examples, which we use to pre-train our models.
Ranked #1 on
Program Synthesis
on SPoC TestW
Explore then Execute: Adapting without Rewards via Factorized Meta-Reinforcement Learning
In principle, meta-reinforcement learning approaches can exploit this shared structure, but in practice, they fail to adapt to new environments when adaptation requires targeted exploration (e. g., exploring the cabinets to find ingredients in a new kitchen).
Selective Question Answering under Domain Shift
In this work, we propose the setting of selective question answering under domain shift, in which a QA model is tested on a mixture of in-domain and out-of-domain data, and must answer (i. e., not abstain on) as many questions as possible while maintaining high accuracy.
Composed Fine-Tuning: Freezing Pre-Trained Denoising Autoencoders for Improved Generalization
Empirically, we show that composed fine-tuning improves over standard fine-tuning on two pseudocode-to-code translation datasets (3% and 6% relative).
Concept Bottleneck Models
We seek to learn models that we can interact with using high-level concepts: if the model did not think there was a bone spur in the x-ray, would it still predict severe arthritis?
Learning Abstract Models for Strategic Exploration and Fast Reward Transfer
Model-based reinforcement learning (RL) is appealing because (i) it enables planning and thus more strategic exploration, and (ii) by decoupling dynamics from rewards, it enables fast transfer to new reward functions.
Robustness to Spurious Correlations via Human Annotations
The reliability of machine learning systems critically assumes that the associations between features and labels remain similar between training and test distributions.
Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices
Learning a new task often requires both exploring to gather task-relevant information and exploiting this information to solve the task.
Task-Oriented Dialogue as Dataflow Synthesis
We describe an approach to task-oriented dialogue in which dialogue state is represented as a dataflow graph.
Simplifying Models with Unlabeled Output Data
We focus on prediction problems with high-dimensional outputs that are subject to output validity constraints, e. g. a pseudocode-to-code translation task where the code must compile.
On the Importance of Adaptive Data Collection for Extremely Imbalanced Pairwise Tasks
Many pairwise classification tasks, such as paraphrase detection and open-domain question answering, naturally have extreme label imbalance (e. g., $99. 99\%$ of examples are negatives).
Learning Adaptive Language Interfaces through Decomposition
Our goal is to create an interactive natural language interface that efficiently and reliably learns from users to complete tasks in simulated robotics settings.
The EOS Decision and Length Extrapolation
Extrapolation to unseen sequence lengths is a challenge for neural generative models of language.
RNNs can generate bounded hierarchical languages with optimal memory
Recurrent neural networks empirically generate natural language with high syntactic fidelity.
Enabling certification of verification-agnostic networks via memory-efficient semidefinite programming
In this work, we propose a first-order dual SDP algorithm that (1) requires memory only linear in the total number of network activations, (2) only requires a fixed number of forward/backward passes through the network per iteration.
Selective Classification Can Magnify Disparities Across Groups
In this paper, we find that while selective classification can improve average accuracies, it can simultaneously magnify existing accuracy disparities between various groups within a population, especially in the presence of spurious correlations.
Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases
To facilitate the development of KBQA models with stronger generalization, we construct and release a new large-scale, high-quality dataset with 64, 331 questions, GrailQA, and provide evaluation settings for all three levels of generalization.
Removing Spurious Features can Hurt Accuracy and Affect Groups Disproportionately
The presence of spurious features interferes with the goal of obtaining robust models that perform well across many groups within the population.
In-N-Out: Pre-Training and Self-Training using Auxiliary Information for Out-of-Distribution Robustness
To get the best of both worlds, we introduce In-N-Out, which first trains a model with auxiliary inputs and uses it to pseudolabel all the in-distribution inputs, then pre-trains a model on OOD auxiliary outputs and fine-tunes this model with the pseudolabels (self-training).
WILDS: A Benchmark of in-the-Wild Distribution Shifts
Distribution shifts -- where the training distribution differs from the test distribution -- can substantially degrade the accuracy of machine learning (ML) systems deployed in the wild.
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Fine-tuning is the de facto way to leverage large pretrained language models to perform downstream tasks.
Do Question Answering Modeling Improvements Hold Across Benchmarks?
Do question answering (QA) modeling improvements (e. g., choice of architecture and training procedure) hold consistently across the diverse landscape of QA benchmarks?
QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering
The problem of answering questions using knowledge from pre-trained language models (LMs) and knowledge graphs (KGs) presents two challenges: given a QA context (question and answer choice), methods need to (i) identify relevant knowledge from large KGs, and (ii) perform joint reasoning over the QA context and KG.
Ranked #2 on
Riddle Sense
on RiddleSense
Swords: A Benchmark for Lexical Substitution with Improved Data Coverage and Quality
We release a new benchmark for lexical substitution, the task of finding appropriate substitutes for a target word in a context.
Break-It-Fix-It: Unsupervised Learning for Program Repair
To bridge this gap, we propose a new training approach, Break-It-Fix-It (BIFI), which has two key ideas: (i) we use the critic to check a fixer's output on real bad inputs and add good (fixed) outputs to the training data, and (ii) we train a breaker to generate realistic bad code from good code.
Ranked #1 on
Program Repair
on DeepFix
Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization
For machine learning systems to be reliable, we must understand their performance in unseen, out-of-distribution environments.
Codified audio language modeling learns useful representations for music information retrieval
Relative to representations from conventional MIR models which are pre-trained on tagging, we find that using representations from Jukebox as input features yields 30% stronger performance on average across four MIR tasks: tagging, genre classification, emotion recognition, and key detection.
Ranked #1 on
Emotion Recognition
on Emomusic
Just Train Twice: Improving Group Robustness without Training Group Information
Standard training via empirical risk minimization (ERM) can produce models that achieve high accuracy on average but low accuracy on certain groups, especially in the presence of spurious correlations between the input and label.
Ranked #1 on
Out-of-Distribution Generalization
on ImageNet-W
On the Opportunities and Risks of Foundation Models
AI is undergoing a paradigm shift with the rise of models (e. g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks.
No True State-of-the-Art? OOD Detection Methods are Inconsistent across Datasets
Out-of-distribution detection is an important component of reliable ML systems.
Out-of-Distribution Detection
Out of Distribution (OOD) Detection
LM-Critic: Language Models for Unsupervised Grammatical Error Correction
Training a model for grammatical error correction (GEC) requires a set of labeled ungrammatical / grammatical sentence pairs, but manually annotating such pairs can be expensive.
Ranked #2 on
Grammatical Error Correction
on Unrestricted
Conditional probing: measuring usable information beyond a baseline
Probing experiments investigate the extent to which neural representations make properties -- like part-of-speech -- predictable.
GreaseLM: Graph REASoning Enhanced Language Models
Answering complex questions about textual narratives requires reasoning over both stated context and the world knowledge that underlies it.
Ensembles and Cocktails: Robust Finetuning for Natural Language Generation
When finetuning a pretrained language model for natural language generation tasks, one is currently faced with a tradeoff.
How does Contrastive Pre-training Connect Disparate Domains?
We develop a conceptual model for contrastive learning under domain shifts, where data augmentations form connections between classes and domains that can be far apart.
Calibrated ensembles - a simple way to mitigate ID-OOD accuracy tradeoffs
We often see undesirable tradeoffs in robust machine learning where out-of-distribution (OOD) accuracy is at odds with in-distribution (ID) accuracy.
Fine-Tuning Distorts Pretrained Features and Underperforms Out-of-Distribution
It is well known that fine-tuning leads to better accuracy in-distribution (ID).
Large Language Models Can Be Strong Differentially Private Learners
Differentially Private (DP) learning has seen limited success for building large deep learning models of text, and straightforward attempts at applying Differentially Private Stochastic Gradient Descent (DP-SGD) to NLP tasks have resulted in large performance drops and high computational overhead.
An Explanation of In-context Learning as Implicit Bayesian Inference
At test time, in-context learning occurs when the LM also infers a shared latent concept between examples in a prompt.
LILA: Language-Informed Latent Actions
We introduce Language-Informed Latent Actions (LILA), a framework for learning natural language interfaces in the context of human-robot collaboration.
Extending the WILDS Benchmark for Unsupervised Adaptation
Unlabeled data can be a powerful point of leverage for mitigating these distribution shifts, as it is frequently much more available than labeled data and can often be obtained from distributions beyond the source distribution as well.
CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities
Large language models (LMs) offer unprecedented language generation capabilities and exciting opportunities for interaction design.
GreaseLM: Graph REASoning Enhanced Language Models for Question Answering
Answering complex questions about textual narratives requires reasoning over both stated context and the world knowledge that underlies it.
Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution
However, in this paper, we find that fine-tuning can achieve worse accuracy than linear probing out-of-distribution (OOD) when the pretrained features are good and the distribution shift is large.
LinkBERT: Pretraining Language Models with Document Links
Language model (LM) pretraining can learn various knowledge from text corpora, helping downstream tasks.
Ranked #1 on
Semantic Similarity
on BIOSSES
Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation
We consider unsupervised domain adaptation (UDA), where labeled data from a source domain (e. g., photographs) and unlabeled data from a target domain (e. g., sketches) are used to learn a classifier for the target domain.
Diffusion-LM Improves Controllable Text Generation
Controlling the behavior of language models (LMs) without re-training is a major open problem in natural language generation.
Decentralized Training of Foundation Models in Heterogeneous Environments
Our key technical contribution is a scheduling algorithm that allocates different computational "tasklets" in the training of foundation models to a group of decentralized GPU devices connected by a slow heterogeneous network.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
Emergent Abilities of Large Language Models
Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks.
Insights into Pre-training via Simpler Synthetic Tasks
Second, to our surprise, we find that pre-training on a simple and generic synthetic task defined by the Set function achieves $65\%$ of the benefits, almost matching LIME.
Is a Caption Worth a Thousand Images? A Controlled Study for Representation Learning
The development of CLIP [Radford et al., 2021] has sparked a debate on whether language supervision can result in vision models with more transferable representations than traditional image-only methods.
Calibrated ensembles can mitigate accuracy tradeoffs under distribution shift
We often see undesirable tradeoffs in robust machine learning where out-of-distribution (OOD) accuracy is at odds with in-distribution (ID) accuracy: a robust classifier obtained via specialized techniques such as removing spurious features often has better OOD but worse ID accuracy compared to a standard classifier trained via ERM.
What Can Transformers Learn In-Context? A Case Study of Simple Function Classes
To make progress towards understanding in-context learning, we consider the well-defined problem of training a model to in-context learn a function class (e. g., linear functions): that is, given data derived from some functions in the class, can we train a model to in-context learn "most" functions from this class?
Improving Self-Supervised Learning by Characterizing Idealized Representations
For non-contrastive learning, we use our framework to derive a simple and novel objective.
Are Sample-Efficient NLP Models More Robust?
Recent results in image classification and extractive question answering have observed that pre-trained models trained on less in-distribution data have better out-of-distribution performance.
Deep Bidirectional Language-Knowledge Graph Pretraining
Pretraining a language model (LM) on text has been shown to help various downstream NLP tasks.
Ranked #1 on
Riddle Sense
on RiddleSense
Surgical Fine-Tuning Improves Adaptation to Distribution Shifts
A common approach to transfer learning under distribution shift is to fine-tune the last few layers of a pre-trained model, preserving learned features while also adapting to the new task.
Contrastive Decoding: Open-ended Text Generation as Optimization
We propose contrastive decoding (CD), a reliable decoding approach that optimizes a contrastive objective subject to a plausibility constraint.
Truncation Sampling as Language Model Desmoothing
In this light, truncation algorithms aim to perform desmoothing, estimating a subset of the support of the true distribution.
Holistic Evaluation of Language Models
We present Holistic Evaluation of Language Models (HELM) to improve the transparency of language models.
How do Authors' Perceptions of their Papers Compare with Co-authors' Perceptions and Peer-review Decisions?
In a top-tier computer science conference (NeurIPS 2021) with more than 23, 000 submitting authors and 9, 000 submitted papers, we survey the authors on three questions: (i) their predicted probability of acceptance for each of their papers, (ii) their perceived ranking of their own papers based on scientific contribution, and (iii) the change in their perception about their own papers after seeing the reviews.
Retrieval-Augmented Multimodal Language Modeling
To integrate knowledge in a more scalable and modular way, we propose a retrieval-augmented multimodal model, which enables a base multimodal model (generator) to refer to relevant text and images fetched by a retriever from external memory (e. g., documents on the web).
Ranked #7 on
Image Captioning
on MS COCO
Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?
As the scope of machine learning broadens, we observe a recurring theme of algorithmic monoculture: the same systems, or systems that share components (e. g. training data), are deployed by multiple decision-makers.
Melody transcription via generative pre-training
The combination of generative pre-training and a new dataset for this task results in $77$% stronger performance on melody transcription relative to the strongest available baseline.
Evaluating Human-Language Model Interaction
To evaluate human-LM interaction, we develop a new framework, Human-AI Language-based Interaction Evaluation (HALIE), that defines the components of interactive systems and dimensions to consider when designing evaluation metrics.
Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP
Retrieval-augmented in-context learning has emerged as a powerful approach for addressing knowledge-intensive tasks using frozen language models (LM) and retrieval models (RM).
"No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomy
Instead of discrete turn-taking between a human and robot, LILAC splits agency between the human and robot: language is an input to a learned model that produces a meaningful, low-dimensional control space that the human can use to guide the robot.
Benchmarking Large Language Models for News Summarization
Large language models (LLMs) have shown promise for automatic summarization but the reasons behind their successes are poorly understood.
Evaluating Self-Supervised Learning via Risk Decomposition
Our decomposition consists of four error components: approximation, representation usability, probe generalization, and encoder generalization.
Data Selection for Language Models via Importance Resampling
To measure whether hashed n-gram features preserve the aspects of the data that are relevant to the target, we define KL reduction, a data metric that measures the proximity between the selected pretraining data and the target on some feature space.
Out-of-Domain Robustness via Targeted Augmentations
Models trained on one set of domains often suffer performance drops on unseen domains, e. g., when wildlife monitoring models are deployed in new camera locations.
Language-Driven Representation Learning for Robotics
First, we demonstrate that existing representations yield inconsistent results across these tasks: masked autoencoding approaches pick up on low-level spatial features at the cost of high-level semantics, while contrastive learning approaches capture the opposite.
Improving Representational Continuity via Continued Pretraining
We consider the continual representation learning setting: sequentially pretrain a model $M'$ on tasks $T_1, \ldots, T_T$, and then adapt $M'$ on a small amount of data from each task $T_i$ to check if it has forgotten information from old tasks.
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
As a result, when running OPT-175B on a single 16GB GPU, FlexGen achieves significantly higher throughput compared to state-of-the-art offloading systems, reaching a generation throughput of 1 token/s for the first time with an effective batch size of 144.
Ecosystem Graphs: The Social Footprint of Foundation Models
Foundation models (e. g. ChatGPT, StableDiffusion) pervasively influence society, warranting immediate social attention.
Whose Opinions Do Language Models Reflect?
Language models (LMs) are increasingly being used in open-ended contexts, where the opinions reflected by LMs in response to subjective queries can have a profound impact, both on user satisfaction, as well as shaping the views of society at large.
Generative Agents: Interactive Simulacra of Human Behavior
Believable proxies of human behavior can empower interactive applications ranging from immersive environments to rehearsal spaces for interpersonal communication to prototyping tools.
Evaluating Verifiability in Generative Search Engines
Generative search engines directly generate responses to user queries, along with in-line citations.
Cheaply Evaluating Inference Efficiency Metrics for Autoregressive Transformer APIs
Large language models (LLMs) power many state-of-the-art systems in natural language processing.
DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining
The mixture proportions of pretraining data domains (e. g., Wikipedia, books, web text) greatly affect language model (LM) performance.
PRODIGY: Enabling In-context Learning Over Graphs
In-context learning is the ability of a pretrained model to adapt to novel and diverse downstream tasks by conditioning on prompt examples, without optimizing any parameters.
AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback
As a demonstration of the research possible in AlpacaFarm, we find that methods that use a reward model can substantially improve over supervised fine-tuning and that our reference PPO implementation leads to a +10% improvement in win-rate against Davinci003.
Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training
Given the massive cost of language model pre-training, a non-trivial improvement of the optimization algorithm would lead to a material reduction on the time and cost of training.
Backpack Language Models
We can interpret a sense vector by inspecting its (non-contextual, linear) projection onto the output space, and intervene on these interpretable hooks to change the model's behavior in predictable ways.
Beyond Positive Scaling: How Negation Impacts Scaling Trends of Language Models
Language models have been shown to exhibit positive scaling, where performance improves as models are scaled up in terms of size, compute, or data.
Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experiment
We present the NeurIPS 2021 consistency experiment, a larger-scale variant of the 2014 NeurIPS experiment in which 10% of conference submissions were reviewed by two independent committees to quantify the randomness in the review process.
One-sided Matrix Completion from Two Observations Per Row
We propose a natural algorithm that involves imputing the missing values of the matrix $X^TX$ and show that even with only two observations per row in $X$, we can provably recover $X^TX$ as long as we have at least $\Omega(r^2 d \log d)$ rows, where $r$ is the rank and $d$ is the number of columns.
Anticipatory Music Transformer
We achieve this by interleaving sequences of events and controls, such that controls appear following stopping times in the event sequence.
Just One Byte (per gradient): A Note on Low-Bandwidth Decentralized Language Model Finetuning Using Shared Randomness
Language model training in distributed settings is limited by the communication cost of gradient exchanges.
Lost in the Middle: How Language Models Use Long Contexts
While recent language models have the ability to take long contexts as input, relatively little is known about how well they use longer context.
Ecosystem-level Analysis of Deployed Machine Learning Reveals Homogeneous Outcomes
In practice, the societal impact of machine learning is determined by the surrounding context of machine learning deployments.
Robust Distortion-free Watermarks for Language Models
We generate watermarked text by mapping a sequence of random numbers -- which we compute using a randomized watermark key -- to a sample from the language model.
MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records
The ability of large language models (LLMs) to follow natural language instructions with human-level fluency suggests many opportunities in healthcare to reduce administrative burden and improve quality of care.
