Search Results for author:
Found 45 papers, 26 papers with code
Variational Bayesian Last Layers
We introduce a deterministic variational formulation for training Bayesian last layer neural networks.
Gemini: A Family of Highly Capable Multimodal Models
This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding.
Ranked #1 on
Multi-task Language Understanding
on MMLU
(using extra training data)
Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models
To do so, we investigate a simple self-training method based on expectation-maximization, which we call ReST$^{EM}$, where we (1) generate samples from the model and filter them using binary feedback, (2) fine-tune the model on these samples, and (3) repeat this process a few times.
Kernel Regression with Infinite-Width Neural Networks on Millions of Examples
Using this approach, we study scaling laws of several neural kernels across many orders of magnitude for the CIFAR-5m dataset.
Plex: Towards Reliability using Pretrained Large Model Extensions
A recent trend in artificial intelligence is the use of pretrained models for language and vision tasks, which have achieved extraordinary performance but also puzzling failures.
Pre-training helps Bayesian optimization too
Contrary to a common belief that BO is suited to optimizing black-box functions, it actually requires domain knowledge on characteristics of those functions to deploy BO successfully.
A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awareness
The most popular approaches to estimate predictive uncertainty in deep learning are methods that combine predictions from multiple neural networks, such as Bayesian neural networks (BNNs) and deep ensembles.
Predicting the utility of search spaces for black-box optimization:a simple, budget-aware approach
For example, when tuning hyperparameters for machine learning pipelines on a new problem given a limited budget, one must strike a balance between excluding potentially promising regions and keeping the search space small enough to be tractable.
Sparse MoEs meet Efficient Ensembles
Machine learning models based on the aggregated outputs of submodels, either at the activation or prediction levels, often exhibit strong performance compared to individual models.
Pre-trained Gaussian processes for Bayesian optimization
Contrary to a common expectation that BO is suited to optimizing black-box functions, it actually requires domain knowledge about those functions to deploy BO successfully.
Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning
In this paper we introduce Uncertainty Baselines: high-quality implementations of standard and state-of-the-art deep learning methods on a variety of tasks.
Deep Learning for Bayesian Optimization of Scientific Problems with High-Dimensional Structure
Bayesian optimization (BO) is a popular paradigm for global optimization of expensive black-box functions, but there are many domains where the function is not completely a black-box.
Exploring the Uncertainty Properties of Neural Networks’ Implicit Priors in the Infinite-Width Limit
This gives us a better understanding of the implicit prior NNs place on function space and allows a direct comparison of the calibration of the NNGP and its finite-width analogue.
Combining Ensembles and Data Augmentation can Harm your Calibration
Ensemble methods which average over multiple neural network predictions are a simple approach to improve a model's calibration and robustness.
Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limit
This gives us a better understanding of the implicit prior NNs place on function space and allows a direct comparison of the calibration of the NNGP and its finite-width analogue.
Training independent subnetworks for robust prediction
Recent approaches to efficiently ensemble neural networks have shown that strong robustness and uncertainty performance can be achieved with a negligible gain in parameters over the original network.
Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Procedures
Uncertainty quantification for complex deep learning models is increasingly important as these techniques see growing use in high-stakes, real-world settings.
A Spectral Energy Distance for Parallel Speech Synthesis
Speech synthesis is an important practical generative modeling problem that has seen great progress over the last few years, with likelihood-based autoregressive neural models now outperforming traditional concatenative systems.
Cold Posteriors and Aleatoric Uncertainty
Recent work has observed that one can outperform exact inference in Bayesian neural networks by tuning the "temperature" of the posterior on a validation set (the "cold posterior" effect).
Revisiting One-vs-All Classifiers for Predictive Uncertainty and Out-of-Distribution Detection in Neural Networks
Accurate estimation of predictive uncertainty in modern neural networks is critical to achieve well calibrated predictions and detect out-of-distribution (OOD) inputs.
Out-of-Distribution Detection
Out of Distribution (OOD) Detection
Hyperparameter Ensembles for Robustness and Uncertainty Quantification
Ensembles over neural network weights trained from different random initialization, known as deep ensembles, achieve state-of-the-art accuracy and calibration.
Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift
Using this one line code change, we achieve state-of-the-art on recent covariate shift benchmarks and an mCE of 60. 28\% on the challenging ImageNet-C dataset; to our knowledge, this is the best result for any model that does not incorporate additional data augmentation or modification of the training pipeline.
Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors
Bayesian neural networks (BNNs) demonstrate promising success in improving the robustness and uncertainty quantification of modern deep learning.
Weighting Is Worth the Wait: Bayesian Optimization with Importance Sampling
Casting hyperparameter search as a multi-task Bayesian optimization problem over both hyperparameters and importance sampling design achieves the best of both worlds: by learning a parameterization of IS that trades-off evaluation complexity and quality, we improve upon Bayesian optimization state-of-the-art runtime and final validation error across a variety of datasets and complex neural architectures.
The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks
Variational Bayesian Inference is a popular methodology for approximating posterior distributions over Bayesian neural network weights.
How Good is the Bayes Posterior in Deep Neural Networks Really?
In this work we cast doubt on the current understanding of Bayes posteriors in popular deep neural networks: we demonstrate through careful MCMC sampling that the posterior predictive induced by the Bayes posterior yields systematically worse predictions compared to simpler methods including point estimates obtained from SGD.
Hydra: Preserving Ensemble Diversity for Model Distillation
As a result, the diversity of the ensemble predictions, stemming from each member, is lost.
Refining the variational posterior through iterative optimization
Variational inference (VI) is a popular approach for approximate Bayesian inference that is particularly promising for highly parameterized models such as deep neural networks.
On the Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks
Variational Bayesian Inference is a popular methodology for approximating posterior distributions in Bayesian neural networks.
Likelihood Ratios for Out-of-Distribution Detection
We propose a likelihood ratio method for deep generative models which effectively corrects for these confounding background statistics.
Out-of-Distribution Detection
Out of Distribution (OOD) Detection
Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
Modern machine learning methods including deep learning have achieved great success in predictive accuracy for supervised learning tasks, but may still fall short in giving useful estimates of their predictive {\em uncertainty}.
On the relationship between Normalising Flows and Variational- and Denoising Autoencoders
Normalising Flows (NFs) are a class of likelihood-based generative models that have recently gained popularity.
DPPNet: Approximating Determinantal Point Processes with Deep Networks
Determinantal Point Processes (DPPs) provide an elegant and versatile way to sample sets of items that balance the point-wise quality with the set-wise diversity of selected items.
Avoiding a Tragedy of the Commons in the Peer Review Process
In this position paper, we argue that a tragedy of the commons outcome may be avoided by emphasizing the professional aspects of this service.
Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling
At the same time, advances in approximate Bayesian methods have made posterior approximation for flexible neural network models practical.
Ranked #1 on
Multi-Armed Bandits
on Mushroom
Learning Latent Permutations with Gumbel-Sinkhorn Networks
Permutations and matchings are core building blocks in a variety of latent variable models, as they allow us to align, canonicalize, and sort data.
Spectral Representations for Convolutional Neural Networks
In this work, we demonstrate that, beyond its advantages for efficient computation, the spectral domain also provides a powerful representation in which to model and train convolutional neural networks (CNNs).
Ranked #169 on
Image Classification
on CIFAR-100
Scalable Bayesian Optimization Using Deep Neural Networks
Bayesian optimization is an effective methodology for the global optimization of functions with expensive evaluations.
Ranked #156 on
Image Classification
on CIFAR-10
Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spaces
In practical Bayesian optimization, we must often search over structures with differing numbers of parameters.
Freeze-Thaw Bayesian Optimization
In this paper we develop a dynamic form of Bayesian optimization for machine learning models with the goal of rapidly finding good hyperparameter settings.
Bayesian Optimization with Unknown Constraints
Recent work on Bayesian optimization has shown its effectiveness in global optimization of difficult black-box objective functions.
Input Warping for Bayesian Optimization of Non-stationary Functions
Bayesian optimization has proven to be a highly effective methodology for the global optimization of unknown, expensive and multimodal functions.
A Determinantal Point Process Latent Variable Model for Inhibition in Neural Spiking Data
Point processes are popular models of neural spiking behavior as they provide a statistical distribution over temporal sequences of spikes and help to reveal the complexities underlying a series of recorded action potentials.
Multi-Task Bayesian Optimization
We demonstrate the utility of this new acquisition function by utilizing a small dataset in order to explore hyperparameter settings for a large dataset.
Ranked #93 on
Image Classification
on STL-10
Practical Bayesian Optimization of Machine Learning Algorithms
In this work, we consider the automatic tuning problem within the framework of Bayesian optimization, in which a learning algorithm's generalization performance is modeled as a sample from a Gaussian process (GP).
Ranked #185 on
Image Classification
on CIFAR-10
