Search Results for author:
Found 96 papers, 51 papers with code
RecurrentGemma: Moving Past Transformers for Efficient Open Language Models
We introduce RecurrentGemma, an open language model which uses Google's novel Griffin architecture.
Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models
Recurrent neural networks (RNNs) have fast inference and scale efficiently on long sequences, but they are difficult to train and hard to scale.
Target Score Matching
Denoising Score Matching estimates the score of a noised version of a target distribution by minimizing a regression loss and is widely used to train the popular class of Denoising Diffusion Models.
Particle Denoising Diffusion Sampler
Denoising diffusion models have become ubiquitous for generative modeling.
Implicit Diffusion: Efficient Optimization through Stochastic Sampling
We present a new algorithm to optimize distributions defined implicitly by parameterized stochastic diffusions.
Marginal Density Ratio for Off-Policy Evaluation in Contextual Bandits
Off-Policy Evaluation (OPE) in contextual bandits is crucial for assessing new policies using existing data without costly experimentation.
Diffusion Generative Inverse Design
Inverse design refers to the problem of optimizing the input of an objective function in order to enact a target outcome.
Reinforced Self-Training (ReST) for Language Modeling
Reinforcement learning from human feedback (RLHF) can improve the quality of large language model's (LLM) outputs by aligning them with human preferences.
Nearly $d$-Linear Convergence Bounds for Diffusion Models via Stochastic Localization
We provide the first convergence bounds which are linear in the data dimension (up to logarithmic factors) assuming only finite second moments of the data distribution.
Conformal prediction under ambiguous ground truth
However, in many real-world scenarios, the labels $Y_1,..., Y_n$ are obtained by aggregating expert opinions using a voting procedure, resulting in a one-hot distribution $\mathbb{P}_{vote}^{Y|X}$.
Evaluating AI systems under uncertain ground truth: a case study in dermatology
In contrast, we propose a framework where aggregation is done using a statistical model.
A Unified Framework for U-Net Design and Analysis
U-Nets are a go-to, state-of-the-art neural architecture across numerous tasks for continuous signals on a square such as images and Partial Differential Equations (PDE), however their design and architecture is understudied.
Tree-Based Diffusion Schrödinger Bridge with Applications to Wasserstein Barycenters
In this paper, we consider an entropic version of mOT with a tree-structured quadratic cost, i. e., a function that can be written as a sum of pairwise cost functions between the nodes of a tree.
Error Bounds for Flow Matching Methods
Previous work derived bounds on the approximation error of diffusion models under the stochastic sampling regime, given assumptions on the $L^2$ loss.
Diffusion Schrödinger Bridge Matching
However, while it is desirable in many applications to approximate the deterministic dynamic Optimal Transport (OT) map which admits attractive properties, DDMs and FMMs are not guaranteed to provide transports close to the OT map.
Denoising Diffusion Samplers
Denoising Diffusion Samplers (DDS) are obtained by approximating the corresponding time-reversal.
Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC
In this work, we build upon these ideas using the score-based interpretation of diffusion models, and explore alternative ways to condition, modify, and reuse diffusion models for tasks involving compositional generation and guidance.
SE(3) diffusion model with application to protein backbone generation
The design of novel protein structures remains a challenge in protein engineering for applications across biomedicine and chemistry.
A Multi-Resolution Framework for U-Nets with Applications to Hierarchical VAEs
U-Net architectures are ubiquitous in state-of-the-art deep learning, however their regularisation properties and relationship to wavelets are understudied.
Causal Falsification of Digital Twins
We consider how to assess the accuracy of a digital twin using real-world data.
Particle-Based Score Estimation for State Space Model Learning in Autonomous Driving
Recent methods addressing this problem typically differentiate through time in a particle filter, which requires workarounds to the non-differentiable resampling step, that yield biased or high variance gradient estimates.
Continuous diffusion for categorical data
Diffusion models have quickly become the go-to paradigm for generative modelling of perceptual signals (such as images and sound) through iterative refinement.
From Denoising Diffusions to Denoising Markov Models
We propose a unifying framework generalising this approach to a wide class of spaces and leading to an original extension of score matching.
Maximum Likelihood Learning of Unnormalized Models for Simulation-Based Inference
We introduce two synthetic likelihood methods for Simulation-Based Inference (SBI), to conduct either amortized or targeted inference from experimental observations when a high-fidelity simulator is available.
Categorical SDEs with Simplex Diffusion
Diffusion models typically operate in the standard framework of generative modelling by producing continuously-valued datapoints.
Alpha-divergence Variational Inference Meets Importance Weighted Auto-Encoders: Methodology and Asymptotics
We then provide two complementary theoretical analyses of the VR-IWAE bound and thus of the standard IWAE bound.
Spectral Diffusion Processes
Score-based generative modelling (SGM) has proven to be a very effective method for modelling densities on finite-dimensional spaces.
Generalisation under gradient descent via deterministic PAC-Bayes
We establish disintegrated PAC-Bayesian generalisation bounds for models trained with gradient descent methods or continuous gradient flows.
Score-Based Diffusion meets Annealed Importance Sampling
To obtain an importance sampling estimate of the marginal likelihood, AIS introduces an extended target distribution to reweight the Markov chain proposal.
Riemannian Diffusion Schrödinger Bridge
Our proposed method generalizes Diffusion Schr\"odinger Bridge introduced in \cite{debortoli2021neurips} to the non-Euclidean setting and extends Riemannian score-based models beyond the first time reversal.
An Empirical Study of Implicit Regularization in Deep Offline RL
Also, we empirically identify three phases of learning that explain the impact of implicit regularization on the learning dynamics and found that bootstrapping alone is insufficient to explain the collapse of the effective rank.
Conformal Off-Policy Prediction in Contextual Bandits
Most off-policy evaluation methods for contextual bandits have focused on the expected outcome of a policy, which is estimated via methods that at best provide only asymptotic guarantees.
A Continuous Time Framework for Discrete Denoising Models
We provide the first complete continuous time framework for denoising diffusion models of discrete data.
Towards Learning Universal Hyperparameter Optimizers with Transformers
Meta-learning hyperparameter optimization (HPO) algorithms from prior experiments is a promising approach to improve optimization efficiency over objective functions from a similar distribution.
Chained Generalisation Bounds
This work discusses how to derive upper bounds for the expected generalisation error of supervised learning algorithms by means of the chaining technique.
Conditional Simulation Using Diffusion Schrödinger Bridges
We extend the Schr\"odinger bridge framework to conditional simulation.
On PAC-Bayesian reconstruction guarantees for VAEs
Despite its wide use and empirical successes, the theoretical understanding and study of the behaviour and performance of the variational autoencoder (VAE) have only emerged in the past few years.
Riemannian Score-Based Generative Modelling
Score-based generative models (SGMs) are a powerful class of generative models that exhibit remarkable empirical performance.
Importance Weighting Approach in Kernel Bayes' Rule
We study a nonparametric approach to Bayesian computation via feature means, where the expectation of prior features is updated to yield expected kernel posterior features, based on regression from learned neural net or kernel features of the observations.
Continual Repeated Annealed Flow Transport Monte Carlo
We propose Continual Repeated Annealed Flow Transport Monte Carlo (CRAFT), a method that combines a sequential Monte Carlo (SMC) sampler (itself a generalization of Annealed Importance Sampling) with variational inference using normalizing flows.
COIN++: Neural Compression Across Modalities
Neural compression algorithms are typically based on autoencoders that require specialized encoder and decoder architectures for different data modalities.
NEO: Non Equilibrium Sampling on the Orbits of a Deterministic Transform
Sampling from a complex distribution $\pi$ and approximating its intractable normalizing constant $\mathrm{Z}$ are challenging problems.
Simulating Diffusion Bridges with Score Matching
This is known to be a challenging problem that has received much attention in the last two decades.
Online Variational Filtering and Parameter Learning
We present a variational method for online state estimation and parameter learning in state-space models (SSMs), a ubiquitous class of latent variable models for sequential data.
Conditionally Gaussian PAC-Bayes
Recent studies have empirically investigated different methods to train stochastic neural networks on a classification task by optimising a PAC-Bayesian bound via stochastic gradient descent.
Learning Optimal Conformal Classifiers
However, using CP as a separate processing step after training prevents the underlying model from adapting to the prediction of confidence sets.
The Curse of Depth in Kernel Regime
Recent work by Jacot et al. (2018) has shown that training a neural network of any kind with gradient descent is strongly related to kernel gradient descent in function space with respect to the Neural Tangent Kernel (NTK).
Mitigating Statistical Bias within Differentially Private Synthetic Data
Increasing interest in privacy-preserving machine learning has led to new and evolved approaches for generating private synthetic data from undisclosed real data.
Quantitative Uniform Stability of the Iterative Proportional Fitting Procedure
We establish the uniform in time stability, w. r. t.
Monte Carlo Variational Auto-Encoders
Variational auto-encoders (VAE) are popular deep latent variable models which are trained by maximizing an Evidence Lower Bound (ELBO).
Wide stochastic networks: Gaussian limit and PAC-Bayesian training
The limit of infinite width allows for substantial simplifications in the analytical study of over-parameterised neural networks.
Diffusion Schrödinger Bridge with Applications to Score-Based Generative Modeling
In contrast, solving the Schr\"odinger Bridge problem (SB), i. e. an entropy-regularized optimal transport problem on path spaces, yields diffusions which generate samples from the data distribution in finite time.
On Instrumental Variable Regression for Deep Offline Policy Evaluation
By applying different IV techniques to OPE, we are not only able to recover previously proposed OPE methods such as model-based techniques but also to obtain competitive new techniques.
NEO: Non Equilibrium Sampling on the Orbit of a Deterministic Transform
Sampling from a complex distribution $\pi$ and approximating its intractable normalizing constant Z are challenging problems.
COIN: COmpression with Implicit Neural representations
We propose a new simple approach for image compression: instead of storing the RGB values for each pixel of an image, we store the weights of a neural network overfitted to the image.
Improving Lossless Compression Rates via Monte Carlo Bits-Back Coding
Naively applied, our schemes would require more initial bits than the standard bits-back coder, but we show how to drastically reduce this additional cost with couplings in the latent space.
Differentiable Particle Filtering via Entropy-Regularized Optimal Transport
Particle Filtering (PF) methods are an established class of procedures for performing inference in non-linear state-space models.
Annealed Flow Transport Monte Carlo
Annealed Importance Sampling (AIS) and its Sequential Monte Carlo (SMC) extensions are state-of-the-art methods for estimating normalizing constants of probability distributions.
Generative Models as Distributions of Functions
By treating data points as functions, we can abstract away from the specific type of data we train on and construct models that are agnostic to discretization.
Learning Deep Features in Instrumental Variable Regression
We propose a novel method, deep feature instrumental variable regression (DFIV), to address the case where relations between instruments, treatments, and outcomes may be nonlinear.
Unbiased Gradient Estimation for Variational Auto-Encoders using Coupled Markov Chains
The variational auto-encoder (VAE) is a deep latent variable model that has two neural networks in an autoencoder-like architecture; one of them parameterizes the model's likelihood.
Variational Inference with Continuously-Indexed Normalizing Flows
Continuously-indexed flows (CIFs) have recently achieved improvements over baseline normalizing flows on a variety of density estimation tasks.
Noisy Adaptive Group Testing using Bayesian Sequential Experimental Design
Our goal in this paper is to propose new group testing algorithms that can operate in a noisy setting (tests can be mistaken) to decide adaptively (looking at past results) which groups to test next, with the goal to converge to a good detection, as quickly, and with as few tests as possible.
Robust Pruning at Initialization
Overparameterized Neural Networks (NN) display state-of-the-art performance.
Schrödinger Bridge Samplers
This is achieved by iteratively modifying the transition kernels of the reference Markov chain to obtain a process whose marginal distribution at time $T$ becomes closer to $\pi_T = \pi$, via regression-based approximations of the corresponding iterative proportional fitting recursion.
Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows
We show that normalising flows become pathological when used to model targets whose supports have complicated topologies.
Localised Generative Flows
We argue that flow-based density models based on continuous bijections are limited in their ability to learn target distributions with complicated topologies, and propose localised generative flows (LGFs) to address this problem.
Modular Meta-Learning with Shrinkage
Many real-world problems, including multi-speaker text-to-speech synthesis, can greatly benefit from the ability to meta-learn large models with only a few task-specific components.
Mean-field Behaviour of Neural Tangent Kernel for Deep Neural Networks
Recent work by Jacot et al. (2018) has shown that training a neural network of any kind with gradient descent in parameter space is strongly related to kernel gradient descent in function space with respect to the Neural Tangent Kernel (NTK).
Efficient MCMC Sampling with Dimension-Free Convergence Rate using ADMM-type Splitting
Performing exact Bayesian inference for complex models is computationally intractable.
Augmented Neural ODEs
We show that Neural Ordinary Differential Equations (ODEs) learn representations that preserve the topology of the input space and prove that this implies the existence of functions Neural ODEs cannot represent.
Ranked #21 on
Image Classification
on MNIST
Bernoulli Race Particle Filters
When the weights in a particle filter are not available analytically, standard resampling methods cannot be employed.
On the Impact of the Activation Function on Deep Neural Networks Training
The weight initialization and the activation function of deep neural networks have a crucial impact on the performance of the training procedure.
Unbiased Smoothing using Particle Independent Metropolis-Hastings
We consider the approximation of expectations with respect to the distribution of a latent Markov process given noisy measurements.
Scalable Metropolis-Hastings for Exact Bayesian Inference with Large Datasets
Bayesian inference via standard Markov Chain Monte Carlo (MCMC) methods is too computationally intensive to handle large datasets, since the cost per step usually scales like $\Theta(n)$ in the number of data points $n$.
Hamiltonian Descent Methods
Yet, crucially the kinetic gradient map can be designed to incorporate information about the convex conjugate in a fashion that allows for linear convergence on convex functions that may be non-smooth or non-strongly convex.
Asymptotic Properties of Recursive Maximum Likelihood Estimation in Non-Linear State-Space Models
Using stochastic gradient search and the optimal filter derivative, it is possible to perform recursive (i. e., online) maximum likelihood estimation in a non-linear state-space model.
Hamiltonian Variational Auto-Encoder
However, for this methodology to be practically efficient, it is necessary to obtain low-variance unbiased estimators of the ELBO and its gradients with respect to the parameters of interest.
On the Selection of Initialization and Activation Function for Deep Neural Networks
We complete this analysis by providing quantitative results showing that, for a class of ReLU-like activation functions, the information propagates indeed deeper for an initialization at the edge of chaos.
Clone MCMC: Parallel High-Dimensional Gaussian Gibbs Sampling
We propose a generalized Gibbs sampler algorithm for obtaining samples approximately distributed from a high-dimensional Gaussian distribution.
Asymptotic Bias of Stochastic Gradient Search
Relying on the same results, the asymptotic behavior of the recursive maximum split-likelihood estimation in hidden Markov models is analyzed, too.
Filtering Variational Objectives
When used as a surrogate objective for maximum likelihood estimation in latent variable models, the evidence lower bound (ELBO) produces state-of-the-art results.
Particle Value Functions
The policy gradients of the expected return objective can react slowly to rare rewards.
Piecewise Deterministic Markov Processes for Scalable Monte Carlo on Restricted Domains
Piecewise Deterministic Monte Carlo algorithms enable simulation from a posterior distribution, whilst only needing to access a sub-sample of data at each iteration.
Methodology Computation
Pseudo-Marginal Hamiltonian Monte Carlo
When following a Markov chain Monte Carlo (MCMC) approach to approximate the posterior distribution in this context, one typically either uses MCMC schemes which target the joint posterior of the parameters and some auxiliary latent variables, or pseudo-marginal Metropolis--Hastings (MH) schemes.
Interacting Particle Markov Chain Monte Carlo
We introduce interacting particle Markov chain Monte Carlo (iPMCMC), a PMCMC method based on an interacting pool of standard and conditional sequential Monte Carlo samplers.
Bayesian nonparametric image segmentation using a generalized Swendsen-Wang algorithm
We introduce here a class of Bayesian nonparametric models to address this problem.
The Bouncy Particle Sampler: A Non-Reversible Rejection-Free Markov Chain Monte Carlo Method
We explore and propose several original extensions of an alternative approach introduced recently in Peters and de With (2012) where the target distribution of interest is explored using a continuous-time Markov process.
Methodology Statistics Theory Statistics Theory
Gibbs Flow for Approximate Transport with Applications to Bayesian Computation
Any measurable function $T:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}$ such that $Y=T(X)\sim\pi_{1}$ if $X\sim\pi_{0}$ is called a transport map from $\pi_{0}$ to $\pi_{1}$.
Computation
Expectation Particle Belief Propagation
The computational complexity of our algorithm at each iteration is quadratic in the number of particles.
On Markov chain Monte Carlo methods for tall data
Finally, we have only been able so far to propose subsampling-based methods which display good performance in scenarios where the Bernstein-von Mises approximation of the target posterior distribution is excellent.
Asynchronous Anytime Sequential Monte Carlo
We introduce a new sequential Monte Carlo algorithm we call the particle cascade.
Fast Computation of Wasserstein Barycenters
We present new algorithms to compute the mean of a set of empirical probability measures under the optimal transport metric.
Bayesian Nonparametric Models on Decomposable Graphs
In latent feature models, we associate to each data point a potentially infinite number of binary latent variables indicating the possession of some features and the IBP is a prior distribution on the associated infinite binary matrix.
