Stochastic Optimization
RMSProp
RMSProp is an unpublished adaptive learning rate optimizer proposed by Geoff Hinton. The motivation is that the magnitude of gradients can differ for different weights, and can change during learning, making it hard to choose a single global learning rate. RMSProp tackles this by keeping a moving average of the squared gradient and adjusting the weight updates by this magnitude. The gradient updates are performed as:
$$E\left[g^{2}\right]_{t} = \gamma E\left[g^{2}\right]_{t-1} + \left(1 - \gamma\right) g^{2}_{t}$$
$$\theta_{t+1} = \theta_{t} - \frac{\eta}{\sqrt{E\left[g^{2}\right]_{t} + \epsilon}}g_{t}$$
Hinton suggests $\gamma=0.9$, with a good default for $\eta$ as $0.001$.
Image: Alec Radford
Papers
| Paper | Code | Results | Date | Stars |
|---|
Tasks
| Task | Papers | Share |
|---|---|---|
| Image Classification | 96 | 11.48% |
| Classification | 39 | 4.67% |
| Deep Learning | 37 | 4.43% |
| General Classification | 35 | 4.19% |
| Semantic Segmentation | 31 | 3.71% |
| Object Detection | 30 | 3.59% |
| Reinforcement Learning | 16 | 1.91% |
| Reinforcement Learning (RL) | 12 | 1.44% |
| Prediction | 10 | 1.20% |
Usage Over Time
Components
| Component | Type |
|
|---|---|---|
| 🤖 No Components Found | You can add them if they exist; e.g. Mask R-CNN uses RoIAlign |
