Search Results for author:
Found 97 papers, 46 papers with code
Diffusion Tempering Improves Parameter Estimation with Probabilistic Integrators for Ordinary Differential Equations
Ordinary differential equations (ODEs) are widely used to describe dynamical systems in science, but identifying parameters that explain experimental measurements is challenging.
Position Paper: Bayesian Deep Learning in the Age of Large-Scale AI
In the current landscape of deep learning research, there is a predominant emphasis on achieving high predictive accuracy in supervised tasks involving large image and language datasets.
Sample Path Regularity of Gaussian Processes from the Covariance Kernel
While applications of GPs are myriad, a comprehensive understanding of GP sample paths, i. e. the function spaces over which they define a probability measure, is lacking.
Kronecker-Factored Approximate Curvature for Modern Neural Network Architectures
In this work, we identify two different settings of linear weight-sharing layers which motivate two flavours of K-FAC -- $\textit{expand}$ and $\textit{reduce}$.
Accelerating Generalized Linear Models by Trading off Computation for Uncertainty
Bayesian Generalized Linear Models (GLMs) define a flexible probabilistic framework to model categorical, ordinal and continuous data, and are widely used in practice.
Parallel-in-Time Probabilistic Numerical ODE Solvers
Probabilistic numerical solvers for ordinary differential equations (ODEs) treat the numerical simulation of dynamical systems as problems of Bayesian state estimation.
The Rank-Reduced Kalman Filter: Approximate Dynamical-Low-Rank Filtering In High Dimensions
In this paper, we propose a novel approximate Gaussian filtering and smoothing method which propagates low-rank approximations of the covariance matrices.
Benchmarking Neural Network Training Algorithms
In order to address these challenges, we introduce a new, competitive, time-to-result benchmark using multiple workloads running on fixed hardware, the AlgoPerf: Training Algorithms benchmark.
Probabilistic Exponential Integrators
However, like standard solvers, they suffer performance penalties for certain stiff systems, where small steps are required not for reasons of numerical accuracy but for the sake of stability.
Uncertainty and Structure in Neural Ordinary Differential Equations
As a first contribution, we show that basic and lightweight Bayesian deep learning techniques like the Laplace approximation can be applied to neural ODEs to yield structured and meaningful uncertainty quantification.
Bayesian Numerical Integration with Neural Networks
Bayesian probabilistic numerical methods for numerical integration offer significant advantages over their non-Bayesian counterparts: they can encode prior information about the integrand, and can quantify uncertainty over estimates of an integral.
Physics-Informed Gaussian Process Regression Generalizes Linear PDE Solvers
Crucially, this probabilistic viewpoint allows to (1) quantify the inherent discretization error; (2) propagate uncertainty about the model parameters to the solution; and (3) condition on noisy measurements.
Optimistic Optimization of Gaussian Process Samples
Bayesian optimization is a popular formalism for global optimization, but its computational costs limit it to expensive-to-evaluate functions.
Approximate Bayesian Neural Operators: Uncertainty Quantification for Parametric PDEs
Neural operators are a type of deep architecture that learns to solve (i. e. learns the nonlinear solution operator of) partial differential equations (PDEs).
Posterior and Computational Uncertainty in Gaussian Processes
For any method in this class, we prove (i) convergence of its posterior mean in the associated RKHS, (ii) decomposability of its combined posterior covariance into mathematical and computational covariances, and (iii) that the combined variance is a tight worst-case bound for the squared error between the method's posterior mean and the latent function.
Posterior Refinement Improves Sample Efficiency in Bayesian Neural Networks
We show that the resulting posterior approximation is competitive with even the gold-standard full-batch Hamiltonian Monte Carlo.
Wasserstein t-SNE
We use t-SNE to construct 2D embeddings of the units, based on the matrix of pairwise Wasserstein distances between them.
Discovering Inductive Bias with Gibbs Priors: A Diagnostic Tool for Approximate Bayesian Inference
By reframing the problem in terms of incompatible conditional distributions we arrive at a natural solution: the Gibbs prior.
Fenrir: Physics-Enhanced Regression for Initial Value Problems
We show how probabilistic numerics can be used to convert an initial value problem into a Gauss--Markov process parametrised by the dynamics of the initial value problem.
ProbNum: Probabilistic Numerics in Python
Probabilistic numerical methods (PNMs) solve numerical problems via probabilistic inference.
Linear-Time Probabilistic Solution of Boundary Value Problems
We propose a fast algorithm for the probabilistic solution of boundary value problems (BVPs), which are ordinary differential equations subject to boundary conditions.
Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning
Deep neural networks are prone to overconfident predictions on outliers.
Probabilistic Numerical Method of Lines for Time-Dependent Partial Differential Equations
Thereby, we extend the toolbox of probabilistic programs for differential equation simulation to PDEs.
Probabilistic ODE Solutions in Millions of Dimensions
Probabilistic solvers for ordinary differential equations (ODEs) have emerged as an efficient framework for uncertainty quantification and inference on dynamical systems.
Pick-and-Mix Information Operators for Probabilistic ODE Solvers
Probabilistic numerical solvers for ordinary differential equations compute posterior distributions over the solution of an initial value problem via Bayesian inference.
Preconditioning for Scalable Gaussian Process Hyperparameter Optimization
While preconditioning is well understood in the context of CG, we demonstrate that it can also accelerate convergence and reduce variance of the estimates for the log-determinant and its derivative.
Laplace Redux -- Effortless Bayesian Deep Learning
Bayesian formulations of deep learning have been shown to have compelling theoretical properties and offer practical functional benefits, such as improved predictive uncertainty quantification and model selection.
Being a Bit Frequentist Improves Bayesian Neural Networks
Despite their compelling theoretical properties, Bayesian neural networks (BNNs) tend to perform worse than frequentist methods in classification-based uncertainty quantification (UQ) tasks such as out-of-distribution (OOD) detection.
Probabilistic DAG Search
Exciting contemporary machine learning problems have recently been phrased in the classic formalism of tree search -- most famously, the game of Go.
Linear-Time Probabilistic Solutions of Boundary Value Problems
We propose a fast algorithm for the probabilistic solution of boundary value problems (BVPs), which are ordinary differential equations subject to boundary conditions.
ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure
Curvature in form of the Hessian or its generalized Gauss-Newton (GGN) approximation is valuable for algorithms that rely on a local model for the loss to train, compress, or explain deep networks.
Laplace Redux - Effortless Bayesian Deep Learning
Bayesian formulations of deep learning have been shown to have compelling theoretical properties and offer practical functional benefits, such as improved predictive uncertainty quantification and model selection.
Informed Equation Learning
Our system then utilizes a robust method to learn equations with atomic functions exhibiting singularities, as e. g. logarithm and division.
Laplace Matching for fast Approximate Inference in Latent Gaussian Models
The method can be thought of as a pre-processing step which can be implemented in <5 lines of code and runs in less than a second.
A Probabilistic State Space Model for Joint Inference from Differential Equations and Data
Mechanistic models with differential equations are a key component of scientific applications of machine learning.
A Probabilistically Motivated Learning Rate Adaptation for Stochastic Optimization
Machine learning practitioners invest significant manual and computational resources in finding suitable learning rates for optimization algorithms.
High-Dimensional Gaussian Process Inference with Derivatives
Although it is widely known that Gaussian processes can be conditioned on observations of the gradient, this functionality is of limited use due to the prohibitive computational cost of $\mathcal{O}(N^3 D^3)$ in data points $N$ and dimension $D$.
Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networks
When engineers train deep learning models, they are very much 'flying blind'.
Bayesian Quadrature on Riemannian Data Manifolds
Riemannian manifolds provide a principled way to model nonlinear geometric structure inherent in data.
Descending through a Crowded Valley — Benchmarking Deep Learning Optimizers
(iii) While we can not discern an optimization method clearly dominating across all tested tasks, we identify a significantly reduced subset of specific algorithms and parameter choices that generally lead to competitive results in our experiments.
ResNet After All: Neural ODEs and Their Numerical Solution
If the trained model is supposed to be a flow generated from an ODE, it should be possible to choose another numerical solver with equal or smaller numerical error without loss of performance.
Stable Implementation of Probabilistic ODE Solvers
Probabilistic solvers for ordinary differential equations (ODEs) provide efficient quantification of numerical uncertainty associated with simulation of dynamical systems.
Calibrated Adaptive Probabilistic ODE Solvers
The contraction rate of this error estimate as a function of the solver's step size identifies it as a well-calibrated worst-case error, but its explicit numerical value for a certain step size is not automatically a good estimate of the explicit error.
Self-Tuning Stochastic Optimization with Curvature-Aware Gradient Filtering
Standard first-order stochastic optimization algorithms base their updates solely on the average mini-batch gradient, and it has been shown that tracking additional quantities such as the curvature can help de-sensitize common hyperparameters.
Probabilistic Linear Solvers for Machine Learning
Linear systems are the bedrock of virtually all numerical computation.
Robot Learning with Crash Constraints
We consider failing behaviors as those that violate a constraint and address the problem of learning with crash constraints, where no data is obtained upon constraint violation.
Learnable Uncertainty under Laplace Approximations
Laplace approximations are classic, computationally lightweight means for constructing Bayesian neural networks (BNNs).
An Infinite-Feature Extension for Bayesian ReLU Nets That Fixes Their Asymptotic Overconfidence
We extend finite ReLU BNNs with infinite ReLU features via the GP and show that the resulting model is asymptotically maximally uncertain far away from the data while the BNNs' predictive power is unaffected near the data.
Fixing Asymptotic Uncertainty of Bayesian Neural Networks with Infinite ReLU Features
However, far away from the training data, even Bayesian neural networks (BNNs) can still underestimate uncertainty and thus be overconfident.
ResNet After All? Neural ODEs and Their Numerical Solution
If the trained model is supposed to be a flow generated from an ODE, it should be possible to choose another numerical solver with equal or smaller numerical error without loss of performance.
Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers
Choosing the optimizer is considered to be among the most crucial design decisions in deep learning, and it is not an easy one.
Bayesian ODE Solvers: The Maximum A Posteriori Estimate
The remaining three classes are termed explicit, semi-implicit, and implicit, which are in similarity with the classical notions corresponding to conditions on the vector field, under which the filter update produces a local maximum a posteriori estimate.
Fast Predictive Uncertainty for Classification with Bayesian Deep Networks
We reconsider old work (Laplace Bridge) to construct a Dirichlet approximation of this softmax output distribution, which yields an analytic map between Gaussian distributions in logit space and Dirichlet distributions (the conjugate prior to the Categorical distribution) in the output space.
Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks
These theoretical results validate the usage of last-layer Bayesian approximation and motivate a range of a fidelity-cost trade-off.
Differentiable Likelihoods for Fast Inversion of 'Likelihood-Free' Dynamical Systems
To address this shortcoming, we employ Gaussian ODE filtering (a probabilistic numerical method for ODEs) to construct a local Gaussian approximation to the likelihood.
BackPACK: Packing more into backprop
Automatic differentiation frameworks are optimized for exactly one thing: computing the average mini-batch gradient.
Conjugate Gradients for Kernel Machines
Regularized least-squares (kernel-ridge / Gaussian process) regression is a fundamental algorithm of statistics and machine learning.
Integrals over Gaussians under Linear Domain Constraints
Integrals of linearly constrained multivariate Gaussian densities are a frequent problem in machine learning and statistics, arising in tasks like generalized linear models and Bayesian optimization.
Classified Regression for Bayesian Optimization: Robot Learning with Unknown Penalties
Learning robot controllers by minimizing a black-box objective cost using Bayesian optimization (BO) can be time-consuming and challenging.
Uncertainty Estimates for Ordinal Embeddings
To investigate objects without a describable notion of distance, one can gather ordinal information by asking triplet comparisons of the form "Is object $x$ closer to $y$ or is $x$ closer to $z$?"
Limitations of the Empirical Fisher Approximation for Natural Gradient Descent
Natural gradient descent, which preconditions a gradient descent update with the Fisher information matrix of the underlying statistical model, is a way to capture partial second-order information.
Convergence Guarantees for Adaptive Bayesian Quadrature Methods
Adaptive Bayesian quadrature (ABQ) is a powerful approach to numerical integration that empirically compares favorably with Monte Carlo integration on problems of medium dimensionality (where non-adaptive quadrature is not competitive).
DeepOBS: A Deep Learning Optimizer Benchmark Suite
We suggest routines and benchmarks for stochastic optimization, with special focus on the unique aspects of deep learning, such as stochasticity, tunability and generalization.
Active Probabilistic Inference on Matrices for Pre-Conditioning in Stochastic Optimization
Pre-conditioning is a well-known concept that can significantly improve the convergence of optimization algorithms.
Modular Block-diagonal Curvature Approximations for Feedforward Architectures
We propose a modular extension of backpropagation for the computation of block-diagonal approximations to various curvature matrices of the training objective (in particular, the Hessian, generalized Gauss-Newton, and positive-curvature Hessian).
Fast and Robust Shortest Paths on Manifolds Learned from Data
We propose a fast, simple and robust algorithm for computing shortest paths and distances on Riemannian manifolds learned from data.
Probabilistic Solutions To Ordinary Differential Equations As Non-Linear Bayesian Filtering: A New Perspective
We formulate probabilistic numerical approximations to solutions of ordinary differential equations (ODEs) as problems in Gaussian process (GP) regression with non-linear measurement functions.
Convergence Rates of Gaussian ODE Filters
A recently-introduced class of probabilistic (uncertainty-aware) solvers for ordinary differential equations (ODEs) applies Gaussian (Kalman) filtering to initial value problems.
Gaussian Processes and Kernel Methods: A Review on Connections and Equivalences
This paper is an attempt to bridge the conceptual gaps between researchers working on the two widely used approaches based on positive definite kernels: Bayesian learning or inference using Gaussian processes on the one side, and frequentist kernel methods based on reproducing kernel Hilbert spaces on the other.
Bayesian Filtering for ODEs with Bounded Derivatives
Recently there has been increasing interest in probabilistic solvers for ordinary differential equations (ODEs) that return full probability measures, instead of point estimates, over the solution and can incorporate uncertainty over the ODE at hand, e. g. if the vector field or the initial value is only approximately known or evaluable.
On the Design of LQR Kernels for Efficient Controller Learning
Finding optimal feedback controllers for nonlinear dynamic systems from data is hard.
Probabilistic Active Learning of Functions in Structural Causal Models
We consider the problem of learning the functions computing children from parents in a Structural Causal Model once the underlying causal graph has been identified.
Krylov Subspace Recycling for Fast Iterative Least-Squares in Machine Learning
To alleviate this problem, several linear-time approximations, such as spectral and inducing-point methods, have been suggested and are now in wide use.
Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients
The ADAM optimizer is exceedingly popular in the deep learning community.
Probabilistic Line Searches for Stochastic Optimization
In deterministic optimization, line searches are a standard tool ensuring stability and efficiency.
Early Stopping without a Validation Set
Early stopping is a widely used technique to prevent poor generalization performance when training an over-expressive model by means of gradient-based optimization.
Virtual vs. Real: Trading Off Simulations and Physical Experiments in Reinforcement Learning with Bayesian Optimization
In practice, the parameters of control policies are often tuned manually.
Coupling Adaptive Batch Sizes with Learning Rates
The batch size significantly influences the behavior of the stochastic optimization algorithm, though, since it determines the variance of the gradient estimates.
A probabilistic model for the numerical solution of initial value problems
Like many numerical methods, solvers for initial value problems (IVPs) on ordinary differential equations estimate an analytically intractable quantity, using the results of tractable computations as inputs.
Exact Sampling from Determinantal Point Processes
Determinantal point processes (DPPs) are an important concept in random matrix theory and combinatorics.
Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets
Bayesian optimization has become a successful tool for hyperparameter optimization of machine learning algorithms, such as support vector machines or deep neural networks.
Active Uncertainty Calibration in Bayesian ODE Solvers
There is resurging interest, in statistics and machine learning, in solvers for ordinary differential equations (ODEs) that return probability measures instead of point estimates.
Automatic LQR Tuning Based on Gaussian Process Global Optimization
With this framework, an initial set of controller gains is automatically improved according to a pre-defined performance objective evaluated from experimental data.
Dual Control for Approximate Bayesian Reinforcement Learning
Control of non-episodic, finite-horizon dynamical systems with uncertain dynamics poses a tough and elementary case of the exploration-exploitation trade-off.
Probabilistic Numerics and Uncertainty in Computations
We deliver a call to arms for probabilistic numerical methods: algorithms for numerical tasks, including linear algebra, integration, optimization and solving differential equations, that return uncertainties in their calculations.
Batch Bayesian Optimization via Local Penalization
The approach assumes that the function of interest, $f$, is a Lipschitz continuous function.
Probabilistic Line Searches for Stochastic Optimization
In deterministic optimization, line searches are a standard tool ensuring stability and efficiency.
Incremental Local Gaussian Regression
Locally weighted regression (LWR) was created as a nonparametric method that can approximate a wide range of functions, is computationally efficient, and can learn continually from very large amounts of incrementally collected data.
Sampling for Inference in Probabilistic Models with Fast Bayesian Quadrature
We propose a novel sampling framework for inference in probabilistic models: an active learning approach that converges more quickly (in wall-clock time) than Markov chain Monte Carlo (MCMC) benchmarks.
Probabilistic ODE Solvers with Runge-Kutta Means
We construct a family of probabilistic numerical methods that instead return a Gauss-Markov process defining a probability distribution over the ODE solution.
Probabilistic Interpretation of Linear Solvers
This manuscript proposes a probabilistic framework for algorithms that iteratively solve unconstrained linear problems $Bx = b$ with positive definite $B$ for $x$.
Local Gaussian Regression
Locally weighted regression was created as a nonparametric learning method that is computationally efficient, can learn from very large amounts of data and add data incrementally.
Active Learning of Linear Embeddings for Gaussian Processes
We propose an active learning method for discovering low-dimensional structure in high-dimensional Gaussian process (GP) tasks.
Probabilistic Solutions to Differential Equations and their Application to Riemannian Statistics
We study a probabilistic numerical method for the solution of both boundary and initial value problems that returns a joint Gaussian process posterior over the solution.
The Randomized Dependence Coefficient
We introduce the Randomized Dependence Coefficient (RDC), a measure of non-linear dependence between random variables of arbitrary dimension based on the Hirschfeld-Gebelein-R\'enyi Maximum Correlation Coefficient.
Optimal Reinforcement Learning for Gaussian Systems
The exploration-exploitation trade-off is among the central challenges of reinforcement learning.
Gaussian Probabilities and Expectation Propagation
We consider these unexpected results empirically and theoretically, both for the problem of Gaussian probabilities and for EP more generally.
