Search Results for author: Eric Vanden-Eijnden

Found 19 papers, 5 papers with code

A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networks

no code implementations28 Oct 2022 Zhengdao Chen, Eric Vanden-Eijnden, Joan Bruna

To understand the training dynamics of neural networks (NNs), prior studies have considered the infinite-width mean-field (MF) limit of two-layer NN, establishing theoretical guarantees of its convergence under gradient flow training as well as its approximation and generalization capabilities.

Building Normalizing Flows with Stochastic Interpolants

no code implementations30 Sep 2022 Michael S. Albergo, Eric Vanden-Eijnden

A simple generative model based on a continuous-time normalizing flow between any pair of base and target probability densities is proposed.

Density Estimation

Learning sparse features can lead to overfitting in neural networks

1 code implementation24 Jun 2022 Leonardo Petrini, Francesco Cagnetta, Eric Vanden-Eijnden, Matthieu Wyart

It is widely believed that the success of deep networks lies in their ability to learn a meaningful representation of the features of the data.

Learning Optimal Flows for Non-Equilibrium Importance Sampling

1 code implementation20 Jun 2022 Yu Cao, Eric Vanden-Eijnden

On the theory side, we discuss how to tailor the velocity field to the target and establish general conditions under which the proposed estimator is a perfect estimator with zero-variance.

Probability flow solution of the Fokker-Planck equation

no code implementations9 Jun 2022 Nicholas M. Boffi, Eric Vanden-Eijnden

The method of choice for integrating the time-dependent Fokker-Planck equation in high-dimension is to generate samples from the solution via integration of the associated stochastic differential equation.

On Feature Learning in Neural Networks with Global Convergence Guarantees

no code implementations22 Apr 2022 Zhengdao Chen, Eric Vanden-Eijnden, Joan Bruna

We study the optimization of wide neural networks (NNs) via gradient flow (GF) in setups that allow feature learning while admitting non-asymptotic global convergence guarantees.

On feature learning in shallow and multi-layer neural networks with global convergence guarantees

no code implementations ICLR 2022 Zhengdao Chen, Eric Vanden-Eijnden, Joan Bruna

We study the optimization of over-parameterized shallow and multi-layer neural networks (NNs) in a regime that allows feature learning while admitting non-asymptotic global convergence guarantees.

Efficient Bayesian Sampling Using Normalizing Flows to Assist Markov Chain Monte Carlo Methods

no code implementations ICML Workshop INNF 2021 Marylou Gabrié, Grant M. Rotskoff, Eric Vanden-Eijnden

Normalizing flows can generate complex target distributions and thus show promise in many applications in Bayesian statistics as an alternative or complement to MCMC for sampling posteriors.

Dual Training of Energy-Based Models with Overparametrized Shallow Neural Networks

no code implementations11 Jul 2021 Carles Domingo-Enrich, Alberto Bietti, Marylou Gabrié, Joan Bruna, Eric Vanden-Eijnden

In the feature-learning regime, this dual formulation justifies using a two time-scale gradient ascent-descent (GDA) training algorithm in which one updates concurrently the particles in the sample space and the neurons in the parameter space of the energy.

On Energy-Based Models with Overparametrized Shallow Neural Networks

1 code implementation15 Apr 2021 Carles Domingo-Enrich, Alberto Bietti, Eric Vanden-Eijnden, Joan Bruna

Energy-based models (EBMs) are a simple yet powerful framework for generative modeling.

Sharp Asymptotic Estimates for Expectations, Probabilities, and Mean First Passage Times in Stochastic Systems with Small Noise

no code implementations8 Mar 2021 Tobias Grafke, Tobias Schäfer, Eric Vanden-Eijnden

Freidlin-Wentzell theory of large deviations can be used to compute the likelihood of extreme or rare events in stochastic dynamical systems via the solution of an optimization problem.

Statistical Mechanics Optimization and Control Probability Fluid Dynamics

A Dynamical Central Limit Theorem for Shallow Neural Networks

no code implementations NeurIPS 2020 Zhengdao Chen, Grant M. Rotskoff, Joan Bruna, Eric Vanden-Eijnden

Furthermore, if the mean-field dynamics converges to a measure that interpolates the training data, we prove that the asymptotic deviation eventually vanishes in the CLT scaling.

Active Importance Sampling for Variational Objectives Dominated by Rare Events: Consequences for Optimization and Generalization

1 code implementation11 Aug 2020 Grant M. Rotskoff, Andrew R. Mitchell, Eric Vanden-Eijnden

Deep neural networks, when optimized with sufficient data, provide accurate representations of high-dimensional functions; in contrast, function approximation techniques that have predominated in scientific computing do not scale well with dimensionality.

BIG-bench Machine Learning Learning Theory

Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions

no code implementations NeurIPS 2020 Stefano Sarao Mannelli, Eric Vanden-Eijnden, Lenka Zdeborová

We consider a teacher-student scenario where the teacher has the same structure as the student with a hidden layer of smaller width $m^*\le m$.

Global convergence of neuron birth-death dynamics

no code implementations5 Feb 2019 Grant Rotskoff, Samy Jelassi, Joan Bruna, Eric Vanden-Eijnden

Neural networks with a large number of parameters admit a mean-field description, which has recently served as a theoretical explanation for the favorable training properties of "overparameterized" models.

Parameters as interacting particles: long time convergence and asymptotic error scaling of neural networks

no code implementations NeurIPS 2018 Grant Rotskoff, Eric Vanden-Eijnden

The performance of neural networks on high-dimensional data distributions suggests that it may be possible to parameterize a representation of a given high-dimensional function with controllably small errors, potentially outperforming standard interpolation methods.

Dynamical computation of the density of states and Bayes factors using nonequilibrium importance sampling

2 code implementations28 Sep 2018 Grant M. Rotskoff, Eric Vanden-Eijnden

Nonequilibrium sampling is potentially much more versatile than its equilibrium counterpart, but it comes with challenges because the invariant distribution is not typically known when the dynamics breaks detailed balance.

Statistical Mechanics

Trainability and Accuracy of Neural Networks: An Interacting Particle System Approach

no code implementations2 May 2018 Grant M. Rotskoff, Eric Vanden-Eijnden

We show that, when the number $n$ of units is large, the empirical distribution of the particles descends on a convex landscape towards the global minimum at a rate independent of $n$, with a resulting approximation error that universally scales as $O(n^{-1})$.

BIG-bench Machine Learning

Cannot find the paper you are looking for? You can Submit a new open access paper.