1 code implementation • NeurIPS 2021 • Huy Tuan Pham, Phan-Minh Nguyen
Treatment of the multilayer case has been missing, with the chief difficulty in finding a formulation that captures the stochastic dependency across not only time but also depth.
no code implementations • ICLR 2021 • Huy Tuan Pham, Phan-Minh Nguyen
In the mean field regime, neural networks are appropriately scaled so that as the width tends to infinity, the learning dynamics tends to a nonlinear and nontrivial dynamical limit, known as the mean field limit.
no code implementations • 16 Feb 2021 • Phan-Minh Nguyen
This limiting description reveals an asymptotically precise picture of feature learning by these models: their training dynamics exhibit different phases that correspond to the learning of different principal subspaces of the data, with varying degrees of nonlinear shrinkage dependent on the $\ell_{2}$-regularization and stopping time.
no code implementations • 16 Jun 2020 • Huy Tuan Pham, Phan-Minh Nguyen
In this companion note, we point out that the insights in our previous work can be readily extended to prove a global convergence guarantee for multilayer networks of any depths.
no code implementations • 30 Jan 2020 • Phan-Minh Nguyen, Huy Tuan Pham
Firstly we show that independent and identically distributed initializations cause strong degeneracy effects on the network's learning trajectory when the network's depth is at least four.
no code implementations • ICLR 2019 • Ping Li, Phan-Minh Nguyen
We study the behavior of weight-tied multilayer vanilla autoencoders under the assumption of random weights.
1 code implementation • 7 Feb 2019 • Phan-Minh Nguyen
Can multilayer neural networks -- typically constructed as highly complex structures with many nonlinearly activated neurons across layers -- behave in a non-trivial way that yet simplifies away a major part of their complexities?
no code implementations • 18 Apr 2018 • Song Mei, Andrea Montanari, Phan-Minh Nguyen
Does SGD converge to a global optimum of the risk or only to a local optimum?
