Search Results for author:
Found 12 papers, 8 papers with code
What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysis
In this work, we bridge this gap by providing a fundamental understanding of what distinguishes the Transformer from the other architectures -- grounded in a theoretical comparison of the (loss) Hessian.
Revisiting Scalable Hessian Diagonal Approximations for Applications in Reinforcement Learning
Second-order information is valuable for many applications but challenging to compute.
Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networks
Physics-informed neural networks (PINNs) are infamous for being hard to train.
Lowering PyTorch's Memory Consumption for Selective Differentiation
This information is useful though to reduce memory whenever gradients are requested for a parameter subset, as is the case in many modern fine-tuning tasks.
Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspective
Adaptive gradient optimizers like Adam(W) are the default training algorithms for many deep learning architectures, such as transformers.
Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFAC
Second-order methods such as KFAC can be useful for neural net training.
On the Disconnect Between Theory and Practice of Neural Networks: Limits of the NTK Perspective
Kernel methods are theoretically well-understood and as a result enjoy algorithmic benefits, which can be demonstrated to hold in wide synthetic neural network architectures.
Convolutions and More as Einsum: A Tensor Network Perspective with Advances for Second-Order Methods
Despite their simple intuition, convolutions are more tedious to analyze than dense layers, which complicates the transfer of theoretical and algorithmic ideas to convolutions.
ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure
Curvature in form of the Hessian or its generalized Gauss-Newton (GGN) approximation is valuable for algorithms that rely on a local model for the loss to train, compress, or explain deep networks.
Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networks
When engineers train deep learning models, they are very much 'flying blind'.
BackPACK: Packing more into backprop
Automatic differentiation frameworks are optimized for exactly one thing: computing the average mini-batch gradient.
Modular Block-diagonal Curvature Approximations for Feedforward Architectures
We propose a modular extension of backpropagation for the computation of block-diagonal approximations to various curvature matrices of the training objective (in particular, the Hessian, generalized Gauss-Newton, and positive-curvature Hessian).
