Search Results for author:
Found 8 papers, 2 papers with code
Why Transformers Need Adam: A Hessian Perspective
SGD performs worse than Adam by a significant margin on Transformers, but the reason remains unclear.
Rethinking SIGN Training: Provable Nonconvex Acceleration without First- and Second-Order Gradient Lipschitz
Sign-based stochastic methods have gained attention due to their ability to achieve robust performance despite using only the sign information for parameter updates.
Adam Can Converge Without Any Modification On Update Rules
We point out there is a mismatch between the settings of theory and practice: Reddi et al. 2018 pick the problem after picking the hyperparameters of Adam, i. e., $(\beta_1, \beta_2)$; while practical applications often fix the problem first and then tune $(\beta_1, \beta_2)$.
Efficient-Adam: Communication-Efficient Distributed Adam
Distributed adaptive stochastic gradient methods have been widely used for large-scale nonconvex optimization, such as training deep learning models.
Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Acceleration
Adam is one of the most influential adaptive stochastic algorithms for training deep neural networks, which has been pointed out to be divergent even in the simple convex setting via a few simple counterexamples.
Quantized Adam with Error Feedback
In this paper, we present a distributed variant of adaptive stochastic gradient method for training deep neural networks in the parameter-server model.
A Unified Analysis of AdaGrad with Weighted Aggregation and Momentum Acceleration
Integrating adaptive learning rate and momentum techniques into SGD leads to a large class of efficiently accelerated adaptive stochastic algorithms, such as AdaGrad, RMSProp, Adam, AccAdaGrad, \textit{etc}.
Arbitrary Style Transfer with Deep Feature Reshuffle
We theoretically prove that our new style loss based on reshuffle connects both global and local style losses respectively used by most parametric and non-parametric neural style transfer methods.
