An Adaptive and Momental Bound Method for Stochastic Learning

2 code implementations • 27 Oct 2019 • Jianbang Ding, Xuancheng Ren, Ruixuan Luo, Xu sun

The dynamic learning rate bounds are based on the exponential moving averages of the adaptive learning rates themselves, which smooth out unexpected large learning rates and stabilize the training of deep neural networks.

Stochastic Optimization