Search Results for author:
Found 9 papers, 5 papers with code
Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models
We show that the heavy-tailed class imbalance found in language modeling tasks leads to difficulties in the optimization dynamics.
Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be
This suggests that Adam outperform SGD because it uses a more robust gradient estimate.
Convergence Rates for the MAP of an Exponential Family and Stochastic Mirror Descent -- an Open Problem
We consider the problem of upper bounding the expected log-likelihood sub-optimality of the maximum likelihood estimate (MLE), or a conjugate maximum a posteriori (MAP) for an exponential family, in a non-asymptotic way.
Homeomorphic-Invariance of EM: Non-Asymptotic Convergence in KL Divergence for Exponential Families via Mirror Descent
In this work we first show that for the common setting of exponential family distributions, viewing EM as a mirror descent algorithm leads to convergence rates in Kullback-Leibler (KL) divergence.
Adaptive Gradient Methods Converge Faster with Over-Parameterization (and you can do a line-search)
Under an interpolation assumption, we prove that AMSGrad with a constant step-size and momentum can converge to the minimizer at the faster $O(1/T)$ rate for smooth, convex functions.
Adaptive Gradient Methods Converge Faster with Over-Parameterization (but you should do a line-search)
In this setting, we prove that AMSGrad with constant step-size and momentum converges to the minimizer at a faster $O(1/T)$ rate.
BackPACK: Packing more into backprop
Automatic differentiation frameworks are optimized for exactly one thing: computing the average mini-batch gradient.
Limitations of the Empirical Fisher Approximation for Natural Gradient Descent
Natural gradient descent, which preconditions a gradient descent update with the Fisher information matrix of the underlying statistical model, is a way to capture partial second-order information.
SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient
Uncertainty estimation in large deep-learning models is a computationally challenging task, where it is difficult to form even a Gaussian approximation to the posterior distribution.
