Search Results for author:
Found 3 papers, 2 papers with code
SKFAC: Training Neural Networks With Faster Kronecker-Factored Approximate Curvature
For the fully connected layers, by utilizing the low-rank property of Kronecker factors of Fisher information matrix, our method only requires inverting a small matrix to approximate the curvature with desirable accuracy.
SKFAC:Training Neural Networks with Faster Kronecker-Factored Approximate Curvature
For the fully connected layers, by utilizing the low-rank property of Kronecker factors of Fisher information matrix, our method only requires inverting a small matrix to approximate the curvature with desirable accuracy.
AsymptoticNG: A regularized natural gradient optimization algorithm with look-ahead strategy
Optimizers that further adjust the scale of gradient, such as Adam, Natural Gradient (NG), etc., despite widely concerned and used by the community, are often found poor generalization performance, compared with Stochastic Gradient Descent (SGD).
