Search Results for author:
Found 18 papers, 0 papers with code
Nonlinear spiked covariance matrices and signal propagation in deep neural networks
Many recent works have studied the eigenvalue spectrum of the Conjugate Kernel (CK) defined by the nonlinear feature map of a feedforward neural network.
Convergence of mean-field Langevin dynamics: Time and space discretization, stochastic gradient, and variance reduction
Despite the generality of our results, we achieve an improved convergence rate in both the SGD and SVRG settings when specialized to the standard Langevin dynamics.
Primal and Dual Analysis of Entropic Fictitious Play for Finite-sum Problems
The entropic fictitious play (EFP) is a recently proposed algorithm that minimizes the sum of a convex functional and entropy in the space of measures -- such an objective naturally arises in the optimization of a two-layer neural network in the mean-field regime.
High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representation
We study the first gradient descent step on the first-layer parameters $\boldsymbol{W}$ in a two-layer neural network: $f(\boldsymbol{x}) = \frac{1}{\sqrt{N}}\boldsymbol{a}^\top\sigma(\boldsymbol{W}^\top\boldsymbol{x})$, where $\boldsymbol{W}\in\mathbb{R}^{d\times N}, \boldsymbol{a}\in\mathbb{R}^{N}$ are randomly initialized, and the training objective is the empirical MSE loss: $\frac{1}{n}\sum_{i=1}^n (f(\boldsymbol{x}_i)-y_i)^2$.
Convex Analysis of the Mean Field Langevin Dynamics
In this work, we give a concise and self-contained convergence rate analysis of the mean field Langevin dynamics with respect to the (regularized) objective function in both continuous and discrete time settings.
Understanding the Variance Collapse of SVGD in High Dimensions
Stein variational gradient descent (SVGD) is a deterministic inference algorithm that evolves a set of particles to fit a target distribution.
Particle Stochastic Dual Coordinate Ascent: Exponential convergent algorithm for mean field neural network optimization
We introduce Particle-SDCA, a gradient-based optimization algorithm for two-layer neural networks in the mean field regime that achieves exponential convergence rate in regularized empirical risk minimization.
Particle Dual Averaging: Optimization of Mean Field Neural Network with Global Convergence Rate Analysis
An important application of the proposed method is the optimization of neural network in the mean field regime, which is theoretically attractive due to the presence of nonlinear feature learning, but quantitative convergence rate can be challenging to obtain.
Particle Dual Averaging: Optimization of Mean Field Neural Networks with Global Convergence Rate Analysis
An important application of the proposed method is the optimization of neural network in the mean field regime, which is theoretically attractive due to the presence of nonlinear feature learning, but quantitative convergence rate can be challenging to obtain.
When Does Preconditioning Help or Hurt Generalization?
While second order optimizers such as natural gradient descent (NGD) often speed up optimization, their effect on generalization has been called into question.
On the Optimal Weighted $\ell_2$ Regularization in Overparameterized Linear Regression
Finally, we determine the optimal weighting matrix $\mathbf{\Sigma}_w$ for both the ridgeless ($\lambda\to 0$) and optimally regularized ($\lambda = \lambda_{\rm opt}$) case, and demonstrate the advantage of the weighted objective over standard ridge regression and PCR.
Generalization of Two-layer Neural Networks: An Asymptotic Viewpoint
This paper investigates the generalization properties of two-layer neural networks in high-dimensions, i. e. when the number of samples $n$, features $d$, and neurons $h$ tend to infinity at the same rate.
Towards Characterizing the High-dimensional Bias of Kernel-based Particle Inference Algorithms
Particle-based inference algorithm is a promising method to efficiently generate samples for an intractable target distribution by iteratively updating a set of particles.
Stochastic Runge-Kutta Accelerates Langevin Monte Carlo and Beyond
In this paper, we establish the convergence rate of sampling algorithms obtained by discretizing smooth It\^o diffusions exhibiting fast Wasserstein-$2$ contraction, based on local deviation properties of the integration scheme.
Modeling the Biological Pathology Continuum with HSIC-regularized Wasserstein Auto-encoders
A crucial challenge in image-based modeling of biomedical data is to identify trends and features that separate normality and pathology.
Post Selection Inference with Incomplete Maximum Mean Discrepancy Estimator
In the paper, we propose a post selection inference (PSI) framework for divergence measure, which can select a set of statistically significant features that discriminate two distributions.
Selecting the Best in GANs Family: a Post Selection Inference Framework
"Which Generative Adversarial Networks (GANs) generates the most plausible images?"
"Dependency Bottleneck" in Auto-encoding Architectures: an Empirical Study
To address this issue, we propose to measure the dependency instead of MI between layers in DNNs.
