Search Results for author:
Found 8 papers, 5 papers with code
Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspective
Adaptive gradient optimizers like Adam(W) are the default training algorithms for many deep learning architectures, such as transformers.
Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFAC for Large Neural Nets
Second-order methods for deep learning -- such as KFAC -- can be useful for neural net training.
On the Disconnect Between Theory and Practice of Overparametrized Neural Networks
Our empirical results demonstrate that this is not the case in optimization, uncertainty quantification or continual learning.
Convolutions Through the Lens of Tensor Networks
Despite their simple intuition, convolutions are more tedious to analyze than dense layers, which complicates the generalization of theoretical and algorithmic ideas.
ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure
Curvature in form of the Hessian or its generalized Gauss-Newton (GGN) approximation is valuable for algorithms that rely on a local model for the loss to train, compress, or explain deep networks.
Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networks
When engineers train deep learning models, they are very much 'flying blind'.
BackPACK: Packing more into backprop
Automatic differentiation frameworks are optimized for exactly one thing: computing the average mini-batch gradient.
Modular Block-diagonal Curvature Approximations for Feedforward Architectures
We propose a modular extension of backpropagation for the computation of block-diagonal approximations to various curvature matrices of the training objective (in particular, the Hessian, generalized Gauss-Newton, and positive-curvature Hessian).
