Search Results for author:
Found 6 papers, 4 papers with code
Normalization Layers Are All That Sharpness-Aware Minimization Needs
Sharpness-aware minimization (SAM) was proposed to reduce sharpness of minima and has been shown to enhance generalization performance in various settings.
What can linear interpolation of neural network loss landscapes tell us?
In this paper, we put inferences of this kind to the test, systematically evaluating how linear interpolation and final performance vary when altering the data, choice of initialization, and other optimizer and architecture design choices.
Multirate Training of Neural Networks
We also discuss splitting choices for the neural network parameters which could enhance generalization performance when neural networks are trained from scratch.
Better Training using Weight-Constrained Stochastic Dynamics
We employ constraints to control the parameter space of deep neural networks throughout training.
Constraint-Based Regularization of Neural Networks
We propose a method for efficiently incorporating constraints into a stochastic gradient Langevin framework for the training of deep neural networks.
Partitioned integrators for thermodynamic parameterization of neural networks
We describe easy-to-implement hybrid partitioned numerical algorithms, based on discretized stochastic differential equations, which are adapted to feed-forward neural networks, including a multi-layer Langevin algorithm, AdLaLa (combining the adaptive Langevin and Langevin algorithms) and LOL (combining Langevin and Overdamped Langevin); we examine the convergence of these methods using numerical studies and compare their performance among themselves and in relation to standard alternatives such as stochastic gradient descent and ADAM.
