Search Results for author:
Found 24 papers, 17 papers with code
Meta-Designing Quantum Experiments with Language Models
Artificial Intelligence (AI) has the potential to significantly advance scientific discovery by finding solutions beyond human capabilities.
A Survey on Data Selection for Language Models
A major factor in the recent success of large language models is the use of enormous and ever-growing text datasets for unsupervised pre-training.
Connect Later: Improving Fine-tuning for Robustness with Targeted Augmentations
Models trained on a labeled source domain (e. g., labeled images from wildlife camera traps) often generalize poorly when deployed on an out-of-distribution (OOD) target domain (e. g., images from new camera trap locations).
DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining
The mixture proportions of pretraining data domains (e. g., Wikipedia, books, web text) greatly affect language model (LM) performance.
Reward Design with Language Models
During training, the LLM evaluates an RL agent's behavior against the desired behavior described by the prompt and outputs a corresponding reward signal.
Data Selection for Language Models via Importance Resampling
To measure whether hashed n-gram features preserve the aspects of the data that are relevant to the target, we define KL reduction, a data metric that measures the proximity between the selected pretraining data and the target on some feature space.
Holistic Evaluation of Language Models
We present Holistic Evaluation of Language Models (HELM) to improve the transparency of language models.
Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models
Toward understanding this implicit bias, we prove that SGD with standard mini-batch noise implicitly prefers flatter minima in language models, and empirically observe a strong correlation between flatness and downstream performance among models with the same minimal pre-training loss.
Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation
We consider unsupervised domain adaptation (UDA), where labeled data from a source domain (e. g., photographs) and unlabeled data from a target domain (e. g., sketches) are used to learn a classifier for the target domain.
Extending the WILDS Benchmark for Unsupervised Adaptation
Unlabeled data can be a powerful point of leverage for mitigating these distribution shifts, as it is frequently much more available than labeled data and can often be obtained from distributions beyond the source distribution as well.
An Explanation of In-context Learning as Implicit Bayesian Inference
At test time, in-context learning occurs when the LM also infers a shared latent concept between examples in a prompt.
How does Contrastive Pre-training Connect Disparate Domains?
We develop a conceptual model for contrastive learning under domain shifts, where data augmentations form connections between classes and domains that can be far apart.
Ensembles and Cocktails: Robust Finetuning for Natural Language Generation
When finetuning a pretrained language model for natural language generation tasks, one is currently faced with a tradeoff.
No True State-of-the-Art? OOD Detection Methods are Inconsistent across Datasets
Out-of-distribution detection is an important component of reliable ML systems.
Out-of-Distribution Detection
Out of Distribution (OOD) Detection
On the Opportunities and Risks of Foundation Models
AI is undergoing a paradigm shift with the rise of models (e. g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
The generative model in our analysis is either a Hidden Markov Model (HMM) or an HMM augmented with a latent memory component, motivated by long-term dependencies in natural language.
WILDS: A Benchmark of in-the-Wild Distribution Shifts
Distribution shifts -- where the training distribution differs from the test distribution -- can substantially degrade the accuracy of machine learning (ML) systems deployed in the wild.
In-N-Out: Pre-Training and Self-Training using Auxiliary Information for Out-of-Distribution Robustness
To get the best of both worlds, we introduce In-N-Out, which first trains a model with auxiliary inputs and uses it to pseudolabel all the in-distribution inputs, then pre-trains a model on OOD auxiliary outputs and fine-tunes this model with the pseudolabels (self-training).
Simplifying Models with Unlabeled Output Data
We focus on prediction problems with high-dimensional outputs that are subject to output validity constraints, e. g. a pseudocode-to-code translation task where the code must compile.
Composed Fine-Tuning: Freezing Pre-Trained Denoising Autoencoders for Improved Generalization
Empirically, we show that composed fine-tuning improves over standard fine-tuning on two pseudocode-to-code translation datasets (3% and 6% relative).
Understanding and Mitigating the Tradeoff Between Robustness and Accuracy
In this work, we precisely characterize the effect of augmentation on the standard error in linear regression when the optimal linear predictor has zero standard and robust error.
Adversarial Training Can Hurt Generalization
While adversarial training can improve robust accuracy (against an adversary), it sometimes hurts standard accuracy (when there is no adversary).
Reparameterizable Subset Sampling via Continuous Relaxations
Many machine learning tasks require sampling a subset of items from a collection based on a parameterized distribution.
Semi-supervised Deep Kernel Learning: Regression with Unlabeled Data by Minimizing Predictive Variance
Large amounts of labeled data are typically required to train deep learning models.
