Search Results for author: Tommy Tang

Found 3 papers, 2 papers with code

AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients

8 code implementations NeurIPS 2020 Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar Tatikonda, Nicha Dvornek, Xenophon Papademetris, James S. Duncan

Viewing the exponential moving average (EMA) of the noisy gradient as the prediction of the gradient at the next time step, if the observed gradient greatly deviates from the prediction, we distrust the current observation and take a small step; if the observed gradient is close to the prediction, we trust it and take a large step.

Image Classification Language Modelling

Momentum Centering and Asynchronous Update for Adaptive Gradient Methods

2 code implementations NeurIPS 2021 Juntang Zhuang, Yifan Ding, Tommy Tang, Nicha Dvornek, Sekhar Tatikonda, James S. Duncan

We demonstrate that ACProp has a convergence rate of $O(\frac{1}{\sqrt{T}})$ for the stochastic non-convex case, which matches the oracle rate and outperforms the $O(\frac{logT}{\sqrt{T}})$ rate of RMSProp and Adam.

Image Classification

Cannot find the paper you are looking for? You can Submit a new open access paper.