Stochastic Optimization

NADAM

NADAM, or Nesterov-accelerated Adaptive Moment Estimation, combines Adam and Nesterov Momentum. The update rule is of the form:

$$ \theta_{t+1} = \theta_{t} - \frac{\eta}{\sqrt{\hat{v}_{t}}+\epsilon}\left(\beta_{1}\hat{m}_{t} + \frac{(1-\beta_{t})g_{t}}{1-\beta^{t}_{1}}\right)$$

Image Source: Incorporating Nesterov Momentum into Adam

Papers


Paper Code Results Date Stars

Tasks


Task Papers Share
Image Classification 1 50.00%
Object Detection 1 50.00%

Components


Component Type
🤖 No Components Found You can add them if they exist; e.g. Mask R-CNN uses RoIAlign

Categories