Search Results for author:
Found 35 papers, 12 papers with code
Optimizing Noise Schedules of Generative Models in High Dimensionss
Recent works have shown that diffusion models can undergo phase transitions, the resolution of which is needed for accurately generating samples.
Model-free learning of probability flows: Elucidating the nonequilibrium dynamics of flocking
Active systems comprise a class of nonequilibrium dynamics in which individual components autonomously dissipate energy.
A Simulation-Free Deep Learning Approach to Stochastic Optimal Control
We propose a simulation-free algorithm for the solution of generic problems in stochastic optimal control (SOC).
NETS: A Non-Equilibrium Transport Sampler
We propose an algorithm, termed the Non-Equilibrium Transport Sampler (NETS), to sample from unnormalized probability distributions.
Flow Map Matching
Generative models based on dynamical transport of measure, such as diffusion models, flow matching models, and stochastic interpolants, learn an ordinary or stochastic differential equation whose trajectories push initial conditions from a known base distribution onto the target.
Sequential-in-time training of nonlinear parametrizations for solving time-dependent partial differential equations
Sequential-in-time methods solve a sequence of training problems to fit nonlinear parametrizations such as neural networks to approximate solution trajectories of partial differential equations over time.
Probabilistic Forecasting with Stochastic Interpolants and Föllmer Processes
We propose a framework for probabilistic forecasting of dynamical systems based on generative modeling.
SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers
We present Scalable Interpolant Transformers (SiT), a family of generative models built on the backbone of Diffusion Transformers (DiT).
Learning to Sample Better
These lecture notes provide an introduction to recent advances in generative modeling methods based on the dynamical transportation of measures, by means of which samples from a simple base measure are mapped to samples from a target measure of interest.
Multimarginal generative modeling with stochastic interpolants
Given a set of $K$ probability densities, we consider the multimarginal generative modeling problem of learning a joint distribution that recovers these densities as marginals.
Stochastic interpolants with data-dependent couplings
In this work, using the framework of stochastic interpolants, we formalize how to \textit{couple} the base and the target densities, whereby samples from the base are computed conditionally given samples from the target in a way that is different from (but does preclude) incorporating information about class labels or continuous embeddings.
Analysis of learning a flow-based generative model from limited sample complexity
We study the problem of training a flow-based generative model, parametrized by a two-layer autoencoder, to sample from a high-dimensional Gaussian mixture.
Deep learning probability flows and entropy production rates in active matter
We show that a single network trained on a system of 4096 particles at one packing fraction can generalize to other regions of the phase diagram, including systems with as many as 32768 particles.
Coupling parameter and particle dynamics for adaptive sampling in Neural Galerkin schemes
Training nonlinear parametrizations such as deep neural networks to numerically approximate solutions of partial differential equations is often based on minimizing a loss that includes the residual, which is analytically available in limited settings only.
Efficient Training of Energy-Based Models Using Jarzynski Equality
Energy-based models (EBMs) are generative models inspired by statistical physics with a wide range of applications in unsupervised learning.
Stochastic Interpolants: A Unifying Framework for Flows and Diffusions
The time-dependent probability density function of the stochastic interpolant is shown to satisfy a first-order transport equation as well as a family of forward and backward Fokker-Planck equations with tunable diffusion coefficient.
A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networks
To understand the training dynamics of neural networks (NNs), prior studies have considered the infinite-width mean-field (MF) limit of two-layer NN, establishing theoretical guarantees of its convergence under gradient flow training as well as its approximation and generalization capabilities.
Building Normalizing Flows with Stochastic Interpolants
A generative model based on a continuous-time normalizing flow between any pair of base and target probability densities is proposed.
Learning sparse features can lead to overfitting in neural networks
It is widely believed that the success of deep networks lies in their ability to learn a meaningful representation of the features of the data.
Learning Optimal Flows for Non-Equilibrium Importance Sampling
On the theory side, we discuss how to tailor the velocity field to the target and establish general conditions under which the proposed estimator is a perfect estimator with zero-variance.
Probability flow solution of the Fokker-Planck equation
The method of choice for integrating the time-dependent Fokker-Planck equation in high-dimension is to generate samples from the solution via integration of the associated stochastic differential equation.
On Feature Learning in Neural Networks with Global Convergence Guarantees
We study the optimization of wide neural networks (NNs) via gradient flow (GF) in setups that allow feature learning while admitting non-asymptotic global convergence guarantees.
Neural Galerkin Schemes with Active Learning for High-Dimensional Evolution Equations
Neural Galerkin schemes build on the Dirac-Frenkel variational principle to train networks by minimizing the residual sequentially over time, which enables adaptively collecting new training data in a self-informed manner that is guided by the dynamics described by the partial differential equations.
On feature learning in shallow and multi-layer neural networks with global convergence guarantees
We study the optimization of over-parameterized shallow and multi-layer neural networks (NNs) in a regime that allows feature learning while admitting non-asymptotic global convergence guarantees.
Efficient Bayesian Sampling Using Normalizing Flows to Assist Markov Chain Monte Carlo Methods
Normalizing flows can generate complex target distributions and thus show promise in many applications in Bayesian statistics as an alternative or complement to MCMC for sampling posteriors.
Dual Training of Energy-Based Models with Overparametrized Shallow Neural Networks
In the feature-learning regime, this dual formulation justifies using a two time-scale gradient ascent-descent (GDA) training algorithm in which one updates concurrently the particles in the sample space and the neurons in the parameter space of the energy.
On Energy-Based Models with Overparametrized Shallow Neural Networks
Energy-based models (EBMs) are a simple yet powerful framework for generative modeling.
Sharp Asymptotic Estimates for Expectations, Probabilities, and Mean First Passage Times in Stochastic Systems with Small Noise
Freidlin-Wentzell theory of large deviations can be used to compute the likelihood of extreme or rare events in stochastic dynamical systems via the solution of an optimization problem.
Statistical Mechanics Optimization and Control Probability Fluid Dynamics
A Dynamical Central Limit Theorem for Shallow Neural Networks
Furthermore, if the mean-field dynamics converges to a measure that interpolates the training data, we prove that the asymptotic deviation eventually vanishes in the CLT scaling.
Active Importance Sampling for Variational Objectives Dominated by Rare Events: Consequences for Optimization and Generalization
Deep neural networks, when optimized with sufficient data, provide accurate representations of high-dimensional functions; in contrast, function approximation techniques that have predominated in scientific computing do not scale well with dimensionality.
Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions
We consider a teacher-student scenario where the teacher has the same structure as the student with a hidden layer of smaller width $m^*\le m$.
Global convergence of neuron birth-death dynamics
Neural networks with a large number of parameters admit a mean-field description, which has recently served as a theoretical explanation for the favorable training properties of "overparameterized" models.
Parameters as interacting particles: long time convergence and asymptotic error scaling of neural networks
The performance of neural networks on high-dimensional data distributions suggests that it may be possible to parameterize a representation of a given high-dimensional function with controllably small errors, potentially outperforming standard interpolation methods.
Dynamical computation of the density of states and Bayes factors using nonequilibrium importance sampling
Nonequilibrium sampling is potentially much more versatile than its equilibrium counterpart, but it comes with challenges because the invariant distribution is not typically known when the dynamics breaks detailed balance.
Statistical Mechanics
Trainability and Accuracy of Neural Networks: An Interacting Particle System Approach
We show that, when the number $n$ of units is large, the empirical distribution of the particles descends on a convex landscape towards the global minimum at a rate independent of $n$, with a resulting approximation error that universally scales as $O(n^{-1})$.
