Search Results for author:
Found 10 papers, 0 papers with code
Sampling from the Mean-Field Stationary Distribution
We study the complexity of sampling from the stationary distribution of a mean-field SDE, or equivalently, the complexity of minimizing a functional over the space of probability measures which includes an interaction term.
Differential Equation Scaling Limits of Shaped and Unshaped Neural Networks
Secondly, for an unshaped MLP at initialization, we derive the first order asymptotic correction to the layerwise correlation.
Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit
We provide experiments demonstrating that residual architectures including convolutional ResNets and Vision Transformers trained with this parameterization exhibit transfer of optimal hyperparameters across width and depth on CIFAR-10 and ImageNet.
The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limit
Motivated by the success of Transformers, we study the covariance matrix of a modified Softmax-based attention model with skip connections in the proportional limit of infinite-depth-and-width.
Improved Discretization Analysis for Underdamped Langevin Monte Carlo
As a byproduct, we also obtain the first KL divergence guarantees for ULMC without Hessian smoothness under strong log-concavity, which is based on a new result on the log-Sobolev constant along the underdamped Langevin diffusion.
The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization
In this work, we study the distribution of this random matrix.
Analysis of Langevin Monte Carlo from Poincaré to Log-Sobolev
Classically, the continuous-time Langevin diffusion converges exponentially fast to its stationary distribution $\pi$ under the sole assumption that $\pi$ satisfies a Poincar\'e inequality.
The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initialization
To provide a better approximation, we study ReLU ResNets in the infinite-depth-and-width limit, where both depth and width tend to infinity as their ratio, $d/n$, remains constant.
Higher Order Generalization Error for First Order Discretization of Langevin Diffusion
We propose a novel approach to analyze generalization error for discretizations of Langevin diffusion, such as the stochastic gradient Langevin dynamics (SGLD).
Riemannian Langevin Algorithm for Solving Semidefinite Programs
We propose a Langevin diffusion-based algorithm for non-convex optimization and sampling on a product manifold of spheres.
