Search Results for author:
Found 10 papers, 2 papers with code
Corridor Geometry in Gradient-Based Optimization
We characterize regions of a loss surface as corridors when the continuous curves of steepest descent -- the solutions of the gradient flow -- become straight lines.
Implicit biases in multitask and continual learning from a backward error analysis perspective
Using backward error analysis, we compute implicit training biases in multitask and continual learning settings for neural networks trained with stochastic gradient descent.
Morse Neural Networks for Uncertainty Quantification
We introduce a new deep generative model useful for uncertainty quantification: the Morse neural network, which generalizes the unnormalized Gaussian densities to have modes of high-dimensional submanifolds instead of just discrete points.
Deep Fusion: Efficient Network Training via Pre-trained Initializations
Finally, we validate our theoretical framework, which guides the optimal use of Deep Fusion, showing that with carefully optimized training dynamics, it significantly reduces both training time and resource consumption.
On a continuous time model of gradient descent dynamics and instability in deep learning
The recipe behind the success of deep learning has been the combination of neural networks and gradient-based optimization.
Why neural networks find simple solutions: the many regularizers of geometric complexity
Using a combination of theoretical arguments and empirical results, we show that many common training heuristics such as parameter norm regularization, spectral norm regularization, flatness regularization, implicit gradient regularization, noise regularization and the choice of parameter initialization all act to control geometric complexity, providing a unifying framework in which to characterize the behavior of deep learning models.
The Geometric Occam's Razor Implicit in Deep Learning
We argue that over-parameterized neural networks trained with stochastic gradient descent are subject to a Geometric Occam's Razor; that is, these networks are implicitly regularized by the geometric model complexity.
Discretization Drift in Two-Player Games
Gradient-based methods for two-player games produce rich dynamics that can solve challenging problems, yet can be difficult to stabilize and understand.
On the Origin of Implicit Regularization in Stochastic Gradient Descent
To interpret this phenomenon we prove that for SGD with random shuffling, the mean SGD iterate also stays close to the path of gradient flow if the learning rate is small and finite, but on a modified loss.
Implicit Gradient Regularization
We call this Implicit Gradient Regularization (IGR) and we use backward error analysis to calculate the size of this regularization.
