Stochastic Optimization
QHM
Introduced by Ma et al. in Quasi-hyperbolic momentum and Adam for deep learningQuasi-Hyperbolic Momentum (QHM) is a stochastic optimization technique that alters momentum SGD with a momentum step, averaging an SGD step with a momentum step:
$$ g_{t+1} = \beta{g_{t}} + \left(1-\beta\right)\cdot{\nabla}\hat{L}_{t}\left(\theta_{t}\right) $$ $$ \theta_{t+1} = \theta_{t} - \alpha\left[\left(1-v\right)\cdot\nabla\hat{L}_{t}\left(\theta_{t}\right) + v\cdot{g_{t+1}}\right]$$
The authors suggest a rule of thumb of $v = 0.7$ and $\beta = 0.999$.
Source: Quasi-hyperbolic momentum and Adam for deep learning
Papers
| Paper | Code | Results | Date | Stars |
|---|
Usage Over Time
Components
| Component | Type |
|
|---|---|---|
| 🤖 No Components Found | You can add them if they exist; e.g. Mask R-CNN uses RoIAlign |
