Search Results for author:
Found 22 papers, 16 papers with code
Holistic Evaluation of Language Models
We present Holistic Evaluation of Language Models (HELM) to improve the transparency of language models.
WILDS: A Benchmark of in-the-Wild Distribution Shifts
Distribution shifts -- where the training distribution differs from the test distribution -- can substantially degrade the accuracy of machine learning (ML) systems deployed in the wild.
On the Opportunities and Risks of Foundation Models
AI is undergoing a paradigm shift with the rise of models (e. g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks.
DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining
The mixture proportions of pretraining data domains (e. g., Wikipedia, books, web text) greatly affect language model (LM) performance.
Reward Design with Language Models
During training, the LLM evaluates an RL agent's behavior against the desired behavior described by the prompt and outputs a corresponding reward signal.
Data Selection for Language Models via Importance Resampling
To measure whether hashed n-gram features preserve the aspects of the data that are relevant to the target, we define KL reduction, a data metric that measures the proximity between the selected pretraining data and the target on some feature space.
An Explanation of In-context Learning as Implicit Bayesian Inference
At test time, in-context learning occurs when the LM also infers a shared latent concept between examples in a prompt.
Semi-supervised Deep Kernel Learning: Regression with Unlabeled Data by Minimizing Predictive Variance
Large amounts of labeled data are typically required to train deep learning models.
A Survey on Data Selection for Language Models
A major factor in the recent success of large language models is the use of enormous and ever-growing text datasets for unsupervised pre-training.
Extending the WILDS Benchmark for Unsupervised Adaptation
Unlabeled data can be a powerful point of leverage for mitigating these distribution shifts, as it is frequently much more available than labeled data and can often be obtained from distributions beyond the source distribution as well.
Reparameterizable Subset Sampling via Continuous Relaxations
Many machine learning tasks require sampling a subset of items from a collection based on a parameterized distribution.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
The generative model in our analysis is either a Hidden Markov Model (HMM) or an HMM augmented with a latent memory component, motivated by long-term dependencies in natural language.
No True State-of-the-Art? OOD Detection Methods are Inconsistent across Datasets
Out-of-distribution detection is an important component of reliable ML systems.
Out-of-Distribution Detection
Out of Distribution (OOD) Detection
Composed Fine-Tuning: Freezing Pre-Trained Denoising Autoencoders for Improved Generalization
Empirically, we show that composed fine-tuning improves over standard fine-tuning on two pseudocode-to-code translation datasets (3% and 6% relative).
Adversarial Training Can Hurt Generalization
While adversarial training can improve robust accuracy (against an adversary), it sometimes hurts standard accuracy (when there is no adversary).
Understanding and Mitigating the Tradeoff Between Robustness and Accuracy
In this work, we precisely characterize the effect of augmentation on the standard error in linear regression when the optimal linear predictor has zero standard and robust error.
In-N-Out: Pre-Training and Self-Training using Auxiliary Information for Out-of-Distribution Robustness
To get the best of both worlds, we introduce In-N-Out, which first trains a model with auxiliary inputs and uses it to pseudolabel all the in-distribution inputs, then pre-trains a model on OOD auxiliary outputs and fine-tunes this model with the pseudolabels (self-training).
How does Contrastive Pre-training Connect Disparate Domains?
We develop a conceptual model for contrastive learning under domain shifts, where data augmentations form connections between classes and domains that can be far apart.
Ensembles and Cocktails: Robust Finetuning for Natural Language Generation
When finetuning a pretrained language model for natural language generation tasks, one is currently faced with a tradeoff.
Simplifying Models with Unlabeled Output Data
We focus on prediction problems with high-dimensional outputs that are subject to output validity constraints, e. g. a pseudocode-to-code translation task where the code must compile.
Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation
We consider unsupervised domain adaptation (UDA), where labeled data from a source domain (e. g., photographs) and unlabeled data from a target domain (e. g., sketches) are used to learn a classifier for the target domain.
Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models
Toward understanding this implicit bias, we prove that SGD with standard mini-batch noise implicitly prefers flatter minima in language models, and empirically observe a strong correlation between flatness and downstream performance among models with the same minimal pre-training loss.
