Search Results for author:
Found 24 papers, 2 papers with code
Understanding Optimal Feature Transfer via a Fine-Grained Bias-Variance Analysis
In the transfer learning paradigm models learn useful representations (or features) during a data-rich pretraining stage, and then use the pretrained representation to improve model performance on data-scarce downstream tasks.
Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models
To do so, we investigate a simple self-training method based on expectation-maximization, which we call ReST$^{EM}$, where we (1) generate samples from the model and filter them using binary feedback, (2) fine-tune the model on these samples, and (3) repeat this process a few times.
Frontier Language Models are not Robust to Adversarial Arithmetic, or "What do I need to say so you agree 2+2=5?
We introduce and study the problem of adversarial arithmetic, which provides a simple yet challenging testbed for language model alignment.
Small-scale proxies for large-scale Transformer training instabilities
In this work, we seek ways to reproduce and study training stability and instability at smaller scales.
Kernel Regression with Infinite-Width Neural Networks on Millions of Examples
Using this approach, we study scaling laws of several neural kernels across many orders of magnitude for the CIFAR-5m dataset.
Ensembling over Classifiers: a Bias-Variance Perspective
Empirically, standard ensembling reducesthe bias, leading us to hypothesize that ensembles of classifiers may perform well in part because of this unexpected reduction. We conclude by an empirical analysis of recent deep learning methods that ensemble over hyperparameters, revealing that these techniques indeed favor bias reduction.
Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions
Stochastic gradient descent (SGD) is a pillar of modern machine learning, serving as the go-to optimization algorithm for a diverse array of problems.
Homogenization of SGD in high-dimensions: Exact dynamics and generalization properties
By analyzing homogenized SGD, we provide exact non-asymptotic high-dimensional expressions for the generalization performance of SGD in terms of a solution of a Volterra integral equation.
Understanding the bias-variance tradeoff of Bregman divergences
We show that, similarly to the label, the central prediction can be interpreted as the mean of a random variable, where the mean operates in a dual space defined by the loss function itself.
Overparameterization Improves Robustness to Covariate Shift in High Dimensions
A significant obstacle in the development of robust machine learning models is \emph{covariate shift}, a form of distribution shift that occurs when the input distributions of the training and test sets differ while the conditional label distributions remain the same.
BIG-bench Machine Learning
Out-of-Distribution Generalization
+1
Covariate Shift in High-Dimensional Random Feature Regression
A significant obstacle in the development of robust machine learning models is covariate shift, a form of distribution shift that occurs when the input distributions of the training and test sets differ while the conditional label distributions remain the same.
BIG-bench Machine Learning
Out-of-Distribution Generalization
+2
Exploring the Uncertainty Properties of Neural Networks’ Implicit Priors in the Infinite-Width Limit
This gives us a better understanding of the implicit prior NNs place on function space and allows a direct comparison of the calibration of the NNGP and its finite-width analogue.
Underspecification Presents Challenges for Credibility in Modern Machine Learning
Predictors returned by underspecified pipelines are often treated as equivalent based on their training domain performance, but we show here that such predictors can behave very differently in deployment domains.
Understanding Double Descent Requires a Fine-Grained Bias-Variance Decomposition
Classical learning theory suggests that the optimal generalization performance of a machine learning model should occur at an intermediate model complexity, with simpler models exhibiting high bias and more complex models exhibiting high variance of the predictive function.
Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limit
This gives us a better understanding of the implicit prior NNs place on function space and allows a direct comparison of the calibration of the NNGP and its finite-width analogue.
The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization
Modern deep learning models employ considerably more parameters than required to fit the training data.
Finite Versus Infinite Neural Networks: an Empirical Study
We perform a careful, thorough, and large scale empirical study of the correspondence between wide neural networks and kernel methods.
Cold Posteriors and Aleatoric Uncertainty
Recent work has observed that one can outperform exact inference in Bayesian neural networks by tuning the "temperature" of the posterior on a validation set (the "cold posterior" effect).
The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks
Modern neural networks are often regarded as complex black-box functions whose behavior is difficult to understand owing to their nonlinear dependence on the data and the nonconvexity in their loss landscapes.
A Random Matrix Perspective on Mixtures of Nonlinearities for Deep Learning
In this work, we focus on this high-dimensional regime in which both the dataset size and the number of features tend to infinity.
Investigating Under and Overfitting in Wasserstein Generative Adversarial Networks
We investigate under and overfitting in Generative Adversarial Networks (GANs), using discriminators unseen by the generator to measure generalization.
Learning GANs and Ensembles Using Discrepancy
Generative adversarial networks (GANs) generate data based on minimizing a divergence between two distributions.
A Random Matrix Perspective on Mixtures of Nonlinearities in High Dimensions
One of the distinguishing characteristics of modern deep learning systems is that they typically employ neural network architectures that utilize enormous numbers of parameters, often in the millions and sometimes even in the billions.
AdaNet: A Scalable and Flexible Framework for Automatically Learning Ensembles
AdaNet is a lightweight TensorFlow-based (Abadi et al., 2015) framework for automatically learning high-quality ensembles with minimal expert intervention.
