Search Results for author:
Found 29 papers, 13 papers with code
Consistent Generative Query Networks
These models typically generate future frames in an autoregressive fashion, which is slow and requires the input and output frames to be consecutive.
Rigorous Agent Evaluation: An Adversarial Approach to Uncover Catastrophic Failures
We demonstrate this is an issue for current agents, where even matching the compute used for training is sometimes insufficient for evaluation.
Consistent Jumpy Predictions for Videos and Scenes
These models typically generate future frames in an autoregressive fashion, which is slow and requires the input and output frames to be consecutive.
Uncovering Surprising Behaviors in Reinforcement Learning via Worst-case Analysis
Reinforcement learning agents are typically trained and evaluated according to their performance averaged over some distribution of environment settings.
Verified Uncertainty Calibration
In these experiments, we also estimate the calibration error and ECE more accurately than the commonly used plugin estimators.
Understanding Self-Training for Gradual Domain Adaptation
Machine learning systems must adapt to data distributions that evolve over time, in applications ranging from sensor networks and self-driving car perception modules to brain-machine interfaces.
Self-training Avoids Using Spurious Features Under Domain Shift
In unsupervised domain adaptation, existing theory focuses on situations where the source and target domains are close.
Selective Classification Can Magnify Disparities Across Groups
In this paper, we find that while selective classification can improve average accuracies, it can simultaneously magnify existing accuracy disparities between various groups within a population, especially in the presence of spurious correlations.
In-N-Out: Pre-Training and Self-Training using Auxiliary Information for Out-of-Distribution Robustness
To get the best of both worlds, we introduce In-N-Out, which first trains a model with auxiliary inputs and uses it to pseudolabel all the in-distribution inputs, then pre-trains a model on OOD auxiliary outputs and fine-tunes this model with the pseudolabels (self-training).
On the Opportunities and Risks of Foundation Models
AI is undergoing a paradigm shift with the rise of models (e. g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks.
No True State-of-the-Art? OOD Detection Methods are Inconsistent across Datasets
Out-of-distribution detection is an important component of reliable ML systems.
Out-of-Distribution Detection
Out of Distribution (OOD) Detection
How does Contrastive Pre-training Connect Disparate Domains?
We develop a conceptual model for contrastive learning under domain shifts, where data augmentations form connections between classes and domains that can be far apart.
Calibrated ensembles - a simple way to mitigate ID-OOD accuracy tradeoffs
We often see undesirable tradeoffs in robust machine learning where out-of-distribution (OOD) accuracy is at odds with in-distribution (ID) accuracy.
Fine-Tuning Distorts Pretrained Features and Underperforms Out-of-Distribution
It is well known that fine-tuning leads to better accuracy in-distribution (ID).
Extending the WILDS Benchmark for Unsupervised Adaptation
Unlabeled data can be a powerful point of leverage for mitigating these distribution shifts, as it is frequently much more available than labeled data and can often be obtained from distributions beyond the source distribution as well.
Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution
However, in this paper, we find that fine-tuning can achieve worse accuracy than linear probing out-of-distribution (OOD) when the pretrained features are good and the distribution shift is large.
Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation
We consider unsupervised domain adaptation (UDA), where labeled data from a source domain (e. g., photographs) and unlabeled data from a target domain (e. g., sketches) are used to learn a classifier for the target domain.
Beyond Separability: Analyzing the Linear Transferability of Contrastive Representations to Related Subpopulations
In particular, a linear classifier trained to separate the representations on the source domain can also predict classes on the target domain accurately, even though the representations of the two domains are far from each other.
Calibrated ensembles can mitigate accuracy tradeoffs under distribution shift
We often see undesirable tradeoffs in robust machine learning where out-of-distribution (OOD) accuracy is at odds with in-distribution (ID) accuracy: a robust classifier obtained via specialized techniques such as removing spurious features often has better OOD but worse ID accuracy compared to a standard classifier trained via ERM.
Are Sample-Efficient NLP Models More Robust?
Recent results in image classification and extractive question answering have observed that pre-trained models trained on less in-distribution data have better out-of-distribution performance.
Surgical Fine-Tuning Improves Adaptation to Distribution Shifts
A common approach to transfer learning under distribution shift is to fine-tune the last few layers of a pre-trained model, preserving learned features while also adapting to the new task.
Holistic Evaluation of Language Models
We present Holistic Evaluation of Language Models (HELM) to improve the transparency of language models.
How to Fine-Tune Vision Models with SGD
SGD and AdamW are the two most used optimizers for fine-tuning large neural networks in computer vision.
Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?
As the scope of machine learning broadens, we observe a recurring theme of algorithmic monoculture: the same systems, or systems that share components (e. g. training data), are deployed by multiple decision-makers.
Finetune like you pretrain: Improved finetuning of zero-shot vision models
In total, these benchmarks establish contrastive finetuning as a simple, intuitive, and state-of-the-art approach for supervised finetuning of image-text models like CLIP.
Improving Representational Continuity via Continued Pretraining
We consider the continual representation learning setting: sequentially pretrain a model $M'$ on tasks $T_1, \ldots, T_T$, and then adapt $M'$ on a small amount of data from each task $T_i$ to check if it has forgotten information from old tasks.
Conservative Prediction via Data-Driven Confidence Minimization
Taking inspiration from this result, we present data-driven confidence minimization (DCM), which minimizes confidence on an uncertainty dataset containing examples that the model is likely to misclassify at test time.
Llamas Know What GPTs Don't Show: Surrogate Models for Confidence Estimation
To maintain user trust, large language models (LLMs) should signal low confidence on examples where they are incorrect, instead of misleading the user.
Evolving Domain Adaptation of Pretrained Language Models for Text Classification
Adapting pre-trained language models (PLMs) for time-series text classification amidst evolving domain shifts (EDS) is critical for maintaining accuracy in applications like stance detection.
