Search Results for author:
Found 113 papers, 86 papers with code
Generating Potent Poisons and Backdoors from Scratch with Guided Diffusion
As a result, we may be able to craft more potent poisons by carefully choosing the base samples.
Mind the GAP: Improving Robustness to Subpopulation Shifts with Group-Aware Priors
Machine learning models often perform poorly under subpopulation shifts in the data distribution.
Chronos: Learning the Language of Time Series
We introduce Chronos, a simple yet effective framework for pretrained probabilistic time series models.
Controllable Prompt Tuning For Balancing Group Distributional Robustness
Models trained on data composed of different groups or domains can suffer from severe performance degradation under distribution shifts.
Fine-Tuned Language Models Generate Stable Inorganic Materials as Text
We propose fine-tuning large language models for generation of stable materials.
Position Paper: Bayesian Deep Learning in the Age of Large-Scale AI
In the current landscape of deep learning research, there is a predominant emphasis on achieving high predictive accuracy in supervised tasks involving large image and language datasets.
Non-Vacuous Generalization Bounds for Large Language Models
Modern language models can contain billions of parameters, raising the question of whether they can generalize beyond the training data or simply regurgitate their training corpora.
Function-Space Regularization in Neural Networks: A Probabilistic Perspective
In this work, we approach regularization in neural networks from a probabilistic perspective and show that by viewing parameter-space regularization as specifying an empirical prior distribution over the model parameters, we can derive a probabilistically well-motivated regularization technique that allows explicitly encoding information about desired predictive functions into neural network training.
Visual Explanations of Image-Text Representations via Multi-Modal Information Bottleneck Attribution
Vision-language pretrained models have seen remarkable success, but their application to safety-critical settings is limited by their lack of interpretability.
Perspectives on the State and Future of Deep Learning -- 2023
The goal of this series is to chronicle opinions and issues in the field of machine learning as they stand today and as they change over time.
Materials Expert-Artificial Intelligence for Materials Discovery
The advent of material databases provides an unprecedented opportunity to uncover predictive descriptors for emergent material properties from vast data space.
Simplifying Neural Network Training Under Class Imbalance
Real-world datasets are often highly class-imbalanced, which can adversely impact the performance of deep learning models.
Should We Learn Most Likely Functions or Parameters?
Moreover, the most likely parameters under the parameter posterior do not generally correspond to the most likely function induced by the parameter posterior.
Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Tasks
Battle of the Backbones (BoB) makes this choice easier by benchmarking a diverse suite of pretrained models, including vision-language models, those trained via self-supervised learning, and the Stable Diffusion backbone, across a diverse set of computer vision tasks ranging from classification to object detection to OOD generalization and more.
Large Language Models Are Zero-Shot Time Series Forecasters
By encoding time series as a string of numerical digits, we can frame time series forecasting as next-token prediction in text.
CoLA: Exploiting Compositional Structure for Automatic and Efficient Numerical Linear Algebra
In this paper, we propose a simple but general framework for large-scale linear algebra problems in machine learning, named CoLA (Compositional Linear Algebra).
Simple and Fast Group Robustness by Automatic Feature Reweighting
A major challenge to out-of-distribution generalization is reliance on spurious features -- patterns that are predictive of the class label in the training data distribution, but not causally related to the target.
User-defined Event Sampling and Uncertainty Quantification in Diffusion Models for Physical Dynamical Systems
In this work, we develop a probabilistic approximation scheme for the conditional score function which provably converges to the true distribution as the noise level decreases.
Protein Design with Guided Discrete Diffusion
A popular approach to protein design is to combine a generative model with a discriminative model for conditional sampling.
A Study of Bayesian Neural Network Surrogates for Bayesian Optimization
Bayesian optimization is a highly efficient approach to optimizing objective functions which are expensive to query.
Automated Few-shot Classification with Instruction-Finetuned Language Models
We observe, in the context of classification tasks, that instruction finetuned language models exhibit remarkable prompt robustness, and we subsequently propose a simple method to eliminate the need for handcrafted prompts, named AuT-Few.
A Stable and Scalable Method for Solving Initial Value PDEs with Neural Networks
Unlike conventional grid and mesh based methods for solving partial differential equations (PDEs), neural networks have the potential to break the curse of dimensionality, providing approximate solutions to problems where using classical solvers is difficult or impossible.
A Cookbook of Self-Supervised Learning
Self-supervised learning, dubbed the dark matter of intelligence, is a promising path to advance machine learning.
The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learning
No free lunch theorems for supervised learning state that no learner can solve all problems or that all learners achieve exactly the same accuracy on average over a uniform distribution on learning problems.
Fortuna: A Library for Uncertainty Quantification in Deep Learning
We present Fortuna, an open-source library for uncertainty quantification in deep learning.
Learning Multimodal Data Augmentation in Feature Space
The ability to jointly learn from multiple modalities, such as text, audio, and visual data, is a defining feature of intelligent systems.
What do Vision Transformers Learn? A Visual Exploration
In addition, we show that ViTs maintain spatial information in all layers except the final layer.
Chroma-VAE: Mitigating Shortcut Learning with Generative Classifiers
Deep neural networks are susceptible to shortcut learning, using simple features to achieve low training loss without discovering essential semantic structure.
PAC-Bayes Compression Bounds So Tight That They Can Explain Generalization
While there has been progress in developing non-vacuous generalization bounds for deep neural networks, these bounds tend to be uninformative about why deep learning works.
K-SAM: Sharpness-Aware Minimization at the Speed of SGD
Sharpness-Aware Minimization (SAM) has recently emerged as a robust technique for improving the accuracy of deep neural networks.
Bayesian Optimization with Conformal Prediction Sets
Bayesian optimization is a coherent, ubiquitous approach to decision-making under uncertainty, with applications including multi-arm bandits, active learning, and black-box optimization.
On Feature Learning in the Presence of Spurious Correlations
Deep classifiers are known to rely on spurious features $\unicode{x2013}$ patterns which are correlated with the target on the training data but not inherently relevant to the learning problem, such as the image backgrounds when classifying the foregrounds.
How Much Data Are Augmentations Worth? An Investigation into Scaling Laws, Invariance, and Implicit Regularization
Despite the clear performance benefits of data augmentations, little is known about why they are so effective.
The Lie Derivative for Measuring Learned Equivariance
In order to better understand the role of equivariance in recent vision models, we introduce the Lie derivative, a method for measuring equivariance with strong mathematical foundations and minimal hyperparameters.
Low-Precision Arithmetic for Fast Gaussian Processes
Low-precision arithmetic has had a transformative effect on the training of neural networks, reducing computation, memory and energy requirements.
Volatility Based Kernels and Moving Average Means for Accurate Forecasting with Gaussian Processes
A broad class of stochastic volatility models are defined by systems of stochastic differential equations.
Transfer Learning with Deep Tabular Models
In this work, we demonstrate that upstream data gives tabular neural networks a decisive advantage over widely used GBDT models.
Low-Precision Stochastic Gradient Langevin Dynamics
While low-precision optimization has been widely used to accelerate deep learning, low-precision sampling remains largely unexplored.
Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priors
Deep learning is increasingly moving towards a transfer learning paradigm whereby large foundation models are fine-tuned on downstream tasks, starting from an initialization learned on the source task.
Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations
Neural network classifiers can largely rely on simple spurious features, such as backgrounds, to make predictions.
Ranked #1 on
Out-of-Distribution Generalization
on UrbanCars
On Uncertainty, Tempering, and Data Augmentation in Bayesian Classification
In Bayesian regression, we often use a Gaussian observation model, where we control the level of aleatoric uncertainty with a noise variance parameter.
Accelerating Bayesian Optimization for Biological Sequence Design with Denoising Autoencoders
Bayesian optimization (BayesOpt) is a gold standard for query-efficient continuous optimization.
Bayesian Model Selection, the Marginal Likelihood, and Generalization
We provide a partial remedy through a conditional marginal likelihood, which we show is more aligned with generalization, and practically valuable for large-scale hyperparameter learning, such as in deep kernel learning.
Deconstructing the Inductive Biases of Hamiltonian Neural Networks
Physics-inspired neural networks (NNs), such as Hamiltonian or Lagrangian NNs, dramatically outperform other learned dynamics models by leveraging strong inductive biases.
When are Iterative Gaussian Processes Reliably Accurate?
While recent work on conjugate gradient methods and Lanczos decompositions have achieved scalable Gaussian process inference with highly accurate point predictions, in several implementations these iterative methods appear to struggle with numerical instabilities in learning kernel hyperparameters, and poor test likelihoods.
Residual Pathway Priors for Soft Equivariance Constraints
There is often a trade-off between building deep learning systems that are expressive enough to capture the nuances of the reality, and having the right inductive biases for efficient learning.
Conditioning Sparse Variational Gaussian Processes for Online Decision-making
With a principled representation of uncertainty and closed form posterior updates, Gaussian processes (GPs) are a natural choice for online decision making.
Bayesian Optimization with High-Dimensional Outputs
However, the Gaussian Process (GP) models typically used as probabilistic surrogates for multi-task Bayesian Optimization scale poorly with the number of outcomes, greatly limiting applicability.
Task-agnostic Continual Learning with Hybrid Probabilistic Models
Learning new tasks continuously without forgetting on a constantly changing data distribution is essential for real-world problems but extremely challenging for modern deep learning.
Dangers of Bayesian Model Averaging under Covariate Shift
Approximate Bayesian inference for neural networks is considered a robust alternative to standard training, often providing good performance on out-of-distribution data.
SKIing on Simplices: Kernel Interpolation on the Permutohedral Lattice for Scalable Gaussian Processes
State-of-the-art methods for scalable Gaussian processes use iterative algorithms, requiring fast matrix vector multiplies (MVMs) with the covariance kernel.
Does Knowledge Distillation Really Work?
Knowledge distillation is a popular technique for training a small student network to emulate a larger teacher model, such as an ensemble of networks.
Scalable Variational Gaussian Processes via Harmonic Kernel Decomposition
We introduce a new scalable variational Gaussian process approximation which provides a high fidelity approximation while retaining general applicability.
What Are Bayesian Neural Network Posteriors Really Like?
The posterior over Bayesian neural network (BNN) parameters is extremely high-dimensional and non-convex.
A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groups
Symmetries and equivariance are fundamental to the generalization of neural networks on domains such as images, graphs, and point clouds.
Kernel Interpolation for Scalable Online Gaussian Processes
Gaussian processes (GPs) provide a gold standard for performance in online settings, such as sample-efficient control and black box optimization, where we need to update a posterior distribution as we acquire data in a sequential fashion.
Fast Adaptation with Linearized Neural Networks
The inductive biases of trained neural networks are difficult to understand and, consequently, to adapt to new settings.
Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling
In this paper, we show that there are mode-connecting simplicial complexes that form multi-dimensional manifolds of low loss, connecting many independently trained models.
Rethinking Parameter Counting: Effective Dimensionality Revisited
Neural networks appear to have mysterious generalization properties when using parameter counting as a proxy for complexity.
Learning Invariances in Neural Networks from Training Data
Invariances to translations have imbued convolutional neural networks with powerful generalization properties.
Simplifying Hamiltonian and Lagrangian Neural Networks via Explicit Constraints
Reasoning about the physical world requires models that are endowed with the right inductive biases to learn the underlying dynamics.
Learning Invariances in Neural Networks
Invariances to translations have imbued convolutional neural networks with powerful generalization properties.
On the model-based stochastic value gradient for continuous reinforcement learning
For over a decade, model-based reinforcement learning has been seen as a way to leverage control-based domain knowledge to improve the sample-efficiency of reinforcement learning agents.
Why Normalizing Flows Fail to Detect Out-of-Distribution Data
Detecting out-of-distribution (OOD) data is crucial for robust machine learning systems.
Improving GAN Training with Probability Ratio Clipping and Sample Reweighting
Despite success on a wide range of problems related to vision, generative adversarial networks (GANs) often suffer from inferior performance due to unstable training, especially for text generation.
Ranked #2 on
Text Generation
on EMNLP2017 WMT
Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited
Neural networks appear to have mysterious generalization properties when using parameter counting as a proxy for complexity.
Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data
The translation equivariance of convolutional layers enables convolutional neural networks to generalize well on image problems.
Bayesian Deep Learning and a Probabilistic Perspective of Generalization
The key distinguishing property of a Bayesian approach is marginalization, rather than using a single setting of weights.
The Case for Bayesian Deep Learning
(3) The structure of neural networks gives rise to a structured prior in function space, which reflects the inductive biases of neural networks that help them generalize.
Towards understanding the true loss surface of deep neural networks using random matrix theory and iterative spectral methods
This approach is an order of magnitude faster than state-of-the-art methods for spectral visualization, and can be generically used to investigate the spectral properties of matrices in deep learning.
Semi-Supervised Learning with Normalizing Flows
Normalizing flows transform a latent distribution through an invertible neural network for a flexible and pleasingly simple approach to generative modelling, while preserving an exact likelihood.
Semi-Supervised Image Classification
Semi-Supervised Text Classification
Randomly Projected Additive Gaussian Processes for Regression
Surprisingly, we find that as the number of random projections increases, the predictive performance of this approach quickly converges to the performance of a kernel operating on the original full dimensional inputs, over a wide range of data sets, even if we are projecting into a single dimension.
Function-Space Distributions over Kernels
The resulting approach enables learning of rich representations, with support for any stationary kernel, uncertainty over the values of the kernel, and an interpretable specification of a prior directly over kernels, without requiring sophisticated initialization or manual intervention.
BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization
Bayesian optimization provides sample-efficient global optimization for a broad range of applications, including automatic machine learning, engineering, physics, and experimental design.
Subspace Inference for Bayesian Deep Learning
Bayesian inference was once a gold standard for learning with neural networks, providing accurate full predictive distributions and well calibrated uncertainty.
Simple Black-box Adversarial Attacks
We propose an intriguingly simple method for the construction of adversarial images in the black-box setting.
SWALP : Stochastic Weight Averaging in Low-Precision Training
Low precision operations can provide scalability, memory savings, portability, and energy efficiency.
MLSys: The New Frontier of Machine Learning Systems
Machine learning (ML) techniques are enjoying rapidly increasing adoption.
Exact Gaussian Processes on a Million Data Points
Gaussian processes (GPs) are flexible non-parametric models, with a capacity that grows with the available data.
Practical Multi-fidelity Bayesian Optimization for Hyperparameter Tuning
Nonetheless, for hyperparameter tuning in deep neural networks, the time required to evaluate the validation error for even a few hyperparameter settings remains a bottleneck.
Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
The posteriors over neural network weights are high dimensional and multimodal.
A Simple Baseline for Bayesian Uncertainty in Deep Learning
We propose SWA-Gaussian (SWAG), a simple, scalable, and general purpose approach for uncertainty representation and calibration in deep learning.
Scaling Gaussian Process Regression with Derivatives
Gaussian processes (GPs) with derivatives are useful in many applications, including Bayesian optimization, implicit surface reconstruction, and terrain reconstruction.
Change Surfaces for Expressive Multidimensional Changepoints and Counterfactual Prediction
We provide a model-agnostic formalization of change surfaces, illustrating how they can provide variable, heterogeneous, and non-monotonic rates of change across multiple dimensions.
GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration
Despite advances in scalable models, the inference tools used for Gaussian processes (GPs) have yet to fully capitalize on developments in computing hardware.
There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average
Presently the most successful approaches to semi-supervised learning are based on consistency regularization, whereby a model is trained to be robust to small perturbations of its inputs and parameters.
Probabilistic FastText for Multi-Sense Word Embeddings
We introduce Probabilistic FastText, a new model for word embeddings that can capture multiple word senses, sub-word structure, and uncertainty information.
Hierarchical Density Order Embeddings
By representing words with probability densities rather than point vectors, probabilistic word embeddings can capture rich and interpretable semantic information and uncertainty.
Ranked #2 on
Lexical Entailment
on HyperLex
Gaussian Process Subset Scanning for Anomalous Pattern Detection in Non-iid Data
We introduce methods for identifying anomalous patterns in non-iid data by combining Gaussian processes with novel log-likelihood ratio statistic and subset scanning techniques.
Constant-Time Predictive Distributions for Gaussian Processes
One of the most compelling features of Gaussian process (GP) regression is its ability to provide well-calibrated posterior distributions.
Averaging Weights Leads to Wider Optima and Better Generalization
Deep neural networks are typically trained by optimizing a loss function with an SGD variant, in conjunction with a decaying learning rate, until convergence.
Ranked #78 on
Image Classification
on CIFAR-100
(using extra training data)
Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
The loss functions of deep neural networks are complex and their geometric properties are not well understood.
Product Kernel Interpolation for Scalable Gaussian Processes
Recent work shows that inference for Gaussian processes can be performed efficiently using iterative methods that rely only on matrix-vector multiplications (MVMs).
Scalable Lévy Process Priors for Spectral Kernel Learning
We propose a distribution over kernels formed by modelling a spectral mixture density with a L\'evy process.
Proceedings of NIPS 2017 Symposium on Interpretable Machine Learning
This is the Proceedings of NIPS 2017 Symposium on Interpretable Machine Learning, held in Long Beach, California, USA on December 7, 2017
Scalable Log Determinants for Gaussian Process Kernel Learning
For applications as varied as Bayesian neural networks, determinantal point processes, elliptical graphical models, and kernel learning for Gaussian processes (GPs), one must compute a log determinant of an $n \times n$ positive definite matrix, and its derivatives - leading to prohibitive $\mathcal{O}(n^3)$ computations.
Bayesian GAN
Generative adversarial networks (GANs) can implicitly learn rich distributions over images, audio, and data which are hard to model with an explicit likelihood.
Multimodal Word Distributions
Word embeddings provide point representations of words containing useful semantic information.
Bayesian Optimization with Gradients
Bayesian optimization has been successful at global optimization of expensive-to-evaluate multimodal objective functions.
Proceedings of NIPS 2016 Workshop on Interpretable Machine Learning for Complex Systems
This is the Proceedings of NIPS 2016 Workshop on Interpretable Machine Learning for Complex Systems, held in Barcelona, Spain on December 9, 2016
Stochastic Variational Deep Kernel Learning
We propose a novel deep kernel learning model and stochastic variational inference procedure which generalizes deep kernel learning approaches to enable classification, multi-task learning, additive covariance structures, and stochastic gradient training.
Learning Scalable Deep Kernels with Recurrent Structure
To model such structure, we propose expressive closed-form kernel functions for Gaussian processes.
Deep Kernel Learning
We introduce scalable deep kernels, which combine the structural properties of deep learning architectures with the non-parametric flexibility of kernel methods.
Thoughts on Massively Scalable Gaussian Processes
This multi-level circulant approximation allows one to unify the orthogonal computational benefits of fast Kronecker and Toeplitz approaches, and is significantly faster than either approach in isolation; 2) local kernel interpolation and inducing points to allow for arbitrarily located data inputs, and $O(1)$ test time predictions; 3) exploiting block-Toeplitz Toeplitz-block structure (BTTB), which enables fast inference and learning when multidimensional Kronecker structure is not present; and 4) projections of the input space to flexibly model correlated inputs and high dimensional data.
The Human Kernel
Bayesian nonparametric models, such as Gaussian processes, provide a compelling framework for automatic statistical modelling: these models have a high degree of flexibility, and automatically calibrated complexity.
Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)
We introduce a new structured kernel interpolation (SKI) framework, which generalises and unifies inducing point methods for scalable Gaussian processes (GPs).
A la Carte - Learning Fast Kernels
Kernel methods have great promise for learning rich statistical representations of large modern datasets.
Student-t Processes as Alternatives to Gaussian Processes
We investigate the Student-t process as an alternative to the Gaussian process as a nonparametric prior over functions.
Bayesian Inference for NMR Spectroscopy with Applications to Chemical Quantification
Nuclear magnetic resonance (NMR) spectroscopy exploits the magnetic properties of atomic nuclei to discover the structure, reaction state and chemical environment of molecules.
GPatt: Fast Multidimensional Pattern Extrapolation with Gaussian Processes
We introduce a new Bayesian nonparametric framework -- GPatt -- enabling automatic pattern extrapolation with Gaussian processes on large multidimensional datasets.
Gaussian Process Kernels for Pattern Discovery and Extrapolation
Gaussian processes are rich distributions over functions, which provide a Bayesian nonparametric approach to smoothing and interpolation.
Gaussian Process Regression Networks
We introduce a new regression framework, Gaussian process regression networks (GPRN), which combines the structural properties of Bayesian neural networks with the non-parametric flexibility of Gaussian processes.
Generalised Wishart Processes
We introduce a stochastic process with Wishart marginals: the generalised Wishart process (GWP).
