Search Results for author:
Found 7 papers, 4 papers with code
Adaptive Strategies in Non-convex Optimization
An algorithm is said to be adaptive to a certain parameter (of the problem) if it does not need a priori knowledge of such a parameter but performs competitively to those that know it.
Robustness to Unbounded Smoothness of Generalized SignSGD
We also compare these algorithms with popular optimizers on a set of deep learning tasks, observing that we can match the performance of Adam while beating the others.
A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networks
Gradient clipping is usually employed to address this issue in the single machine setting, but exploring this technique in the distributed setting is still in its infancy: it remains mysterious whether the gradient clipping scheme can take advantage of multiple machines to enjoy parallel speedup.
Understanding AdamW through Proximal Methods and Scale-Freeness
First, we show how to re-interpret AdamW as an approximation of a proximal gradient method, which takes advantage of the closed-form proximal mapping of the regularizer instead of only utilizing its gradient information as in Adam-$\ell_2$.
A Second look at Exponential and Cosine Step Sizes: Simplicity, Adaptivity, and Performance
Moreover, we show the surprising property that these two strategies are \emph{adaptive} to the noise level in the stochastic gradients of PL functions.
No-regret Non-convex Online Meta-Learning
The online meta-learning framework is designed for the continual lifelong learning setting.
Surrogate Losses for Online Learning of Stepsizes in Stochastic Non-Convex Optimization
Stochastic Gradient Descent (SGD) has played a central role in machine learning.
