Search Results for author:
Found 30 papers, 11 papers with code
Trainability and Accuracy of Neural Networks: An Interacting Particle System Approach
We show that, when the number $n$ of units is large, the empirical distribution of the particles descends on a convex landscape towards the global minimum at a rate independent of $n$, with a resulting approximation error that universally scales as $O(n^{-1})$.
Dynamical computation of the density of states and Bayes factors using nonequilibrium importance sampling
Nonequilibrium sampling is potentially much more versatile than its equilibrium counterpart, but it comes with challenges because the invariant distribution is not typically known when the dynamics breaks detailed balance.
Statistical Mechanics
Parameters as interacting particles: long time convergence and asymptotic error scaling of neural networks
The performance of neural networks on high-dimensional data distributions suggests that it may be possible to parameterize a representation of a given high-dimensional function with controllably small errors, potentially outperforming standard interpolation methods.
Global convergence of neuron birth-death dynamics
Neural networks with a large number of parameters admit a mean-field description, which has recently served as a theoretical explanation for the favorable training properties of "overparameterized" models.
Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions
We consider a teacher-student scenario where the teacher has the same structure as the student with a hidden layer of smaller width $m^*\le m$.
Active Importance Sampling for Variational Objectives Dominated by Rare Events: Consequences for Optimization and Generalization
Deep neural networks, when optimized with sufficient data, provide accurate representations of high-dimensional functions; in contrast, function approximation techniques that have predominated in scientific computing do not scale well with dimensionality.
A Dynamical Central Limit Theorem for Shallow Neural Networks
Furthermore, if the mean-field dynamics converges to a measure that interpolates the training data, we prove that the asymptotic deviation eventually vanishes in the CLT scaling.
Sharp Asymptotic Estimates for Expectations, Probabilities, and Mean First Passage Times in Stochastic Systems with Small Noise
Freidlin-Wentzell theory of large deviations can be used to compute the likelihood of extreme or rare events in stochastic dynamical systems via the solution of an optimization problem.
Statistical Mechanics Optimization and Control Probability Fluid Dynamics
On Energy-Based Models with Overparametrized Shallow Neural Networks
Energy-based models (EBMs) are a simple yet powerful framework for generative modeling.
Dual Training of Energy-Based Models with Overparametrized Shallow Neural Networks
In the feature-learning regime, this dual formulation justifies using a two time-scale gradient ascent-descent (GDA) training algorithm in which one updates concurrently the particles in the sample space and the neurons in the parameter space of the energy.
Efficient Bayesian Sampling Using Normalizing Flows to Assist Markov Chain Monte Carlo Methods
Normalizing flows can generate complex target distributions and thus show promise in many applications in Bayesian statistics as an alternative or complement to MCMC for sampling posteriors.
On feature learning in shallow and multi-layer neural networks with global convergence guarantees
We study the optimization of over-parameterized shallow and multi-layer neural networks (NNs) in a regime that allows feature learning while admitting non-asymptotic global convergence guarantees.
Neural Galerkin Schemes with Active Learning for High-Dimensional Evolution Equations
Neural Galerkin schemes build on the Dirac-Frenkel variational principle to train networks by minimizing the residual sequentially over time, which enables adaptively collecting new training data in a self-informed manner that is guided by the dynamics described by the partial differential equations.
On Feature Learning in Neural Networks with Global Convergence Guarantees
We study the optimization of wide neural networks (NNs) via gradient flow (GF) in setups that allow feature learning while admitting non-asymptotic global convergence guarantees.
Probability flow solution of the Fokker-Planck equation
The method of choice for integrating the time-dependent Fokker-Planck equation in high-dimension is to generate samples from the solution via integration of the associated stochastic differential equation.
Learning Optimal Flows for Non-Equilibrium Importance Sampling
On the theory side, we discuss how to tailor the velocity field to the target and establish general conditions under which the proposed estimator is a perfect estimator with zero-variance.
Learning sparse features can lead to overfitting in neural networks
It is widely believed that the success of deep networks lies in their ability to learn a meaningful representation of the features of the data.
Building Normalizing Flows with Stochastic Interpolants
A generative model based on a continuous-time normalizing flow between any pair of base and target probability densities is proposed.
A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networks
To understand the training dynamics of neural networks (NNs), prior studies have considered the infinite-width mean-field (MF) limit of two-layer NN, establishing theoretical guarantees of its convergence under gradient flow training as well as its approximation and generalization capabilities.
Stochastic Interpolants: A Unifying Framework for Flows and Diffusions
The time-dependent probability density function of the stochastic interpolant is shown to satisfy a first-order transport equation as well as a family of forward and backward Fokker-Planck equations with tunable diffusion coefficient.
Efficient Training of Energy-Based Models Using Jarzynski Equality
Energy-based models (EBMs) are generative models inspired by statistical physics with a wide range of applications in unsupervised learning.
Coupling parameter and particle dynamics for adaptive sampling in Neural Galerkin schemes
Training nonlinear parametrizations such as deep neural networks to numerically approximate solutions of partial differential equations is often based on minimizing a loss that includes the residual, which is analytically available in limited settings only.
Deep learning probability flows and entropy production rates in active matter
We show that a single instance of our network trained on a system of 4096 particles at one packing fraction can generalize to other regions of the phase diagram, including systems with as many as 32768 particles.
Stochastic interpolants with data-dependent couplings
In this work, using the framework of stochastic interpolants, we formalize how to \textit{couple} the base and the target densities, whereby samples from the base are computed conditionally given samples from the target in a way that is different from (but does preclude) incorporating information about class labels or continuous embeddings.
Multimarginal generative modeling with stochastic interpolants
Given a set of $K$ probability densities, we consider the multimarginal generative modeling problem of learning a joint distribution that recovers these densities as marginals.
Analysis of learning a flow-based generative model from limited sample complexity
We study the problem of training a flow-based generative model, parametrized by a two-layer autoencoder, to sample from a high-dimensional Gaussian mixture.
Learning to Sample Better
These lecture notes provide an introduction to recent advances in generative modeling methods based on the dynamical transportation of measures, by means of which samples from a simple base measure are mapped to samples from a target measure of interest.
SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers
We present Scalable Interpolant Transformers (SiT), a family of generative models built on the backbone of Diffusion Transformers (DiT).
Probabilistic Forecasting with Stochastic Interpolants and Föllmer Processes
We propose a framework for probabilistic forecasting of dynamical systems based on generative modeling.
Sequential-in-time training of nonlinear parametrizations for solving time-dependent partial differential equations
Sequential-in-time methods solve a sequence of training problems to fit nonlinear parametrizations such as neural networks to approximate solution trajectories of partial differential equations over time.
