Stochastic Optimization

Edit

General • Optimization • 62 methods

Stochastic Optimization methods are used to optimize neural networks. We typically take a mini-batch of data, hence 'stochastic', and perform a type of gradient descent with this minibatch. Below you can find a continuously updating list of stochastic optimization algorithms.

Subcategories

1 Large Batch Optimization

2 Momentum Rules

Methods

Add a Method

Method	Year	Papers
Adam Adam: A Method for Stochastic Optimization	2014	14639
SGD	1951	1646
Adafactor Adafactor: Adaptive Learning Rates with Sublinear Memory Cost	2018	502
RMSProp	2013	409
LAMB Large Batch Optimization for Deep Learning: Training BERT in 76 minutes	2019	171
AdaGrad	2011	152
SGD with Momentum	1999	137
AdamW Decoupled Weight Decay Regularization	2017	126
Deep Ensembles Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles	2016	122
Gravity Gravity Optimizer: a Kinematic Approach on Optimization in Deep Learning	2021	112
FA Random feedback weights support learning in deep neural networks	2014	81
MAS Mixing ADAM and SGD: a Combined Optimization Method	2020	75
LARS Large Batch Training of Convolutional Networks	2017	61
Local SGD Local SGD Converges Fast and Communicates Little	2018	56
AMSGrad On the Convergence of Adam and Beyond	2019	46
RAdam On the Variance of the Adaptive Learning Rate and Beyond	2019	44
DFA Direct Feedback Alignment Provides Learning in Deep Neural Networks	2016	38
Stochastic Weight Averaging Averaging Weights Leads to Wider Optima and Better Generalization	2018	35
Apollo Apollo: An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimization	2020	29
Gradient Sparsification Gradient Sparsification for Communication-Efficient Distributed Optimization	2017	28
Nesterov Accelerated Gradient	1983	26
1-bit Adam 1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed	2021	26
KP Deep Learning without Weight Transport	2019	21
Adabelief AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients	2020	16
AdaDelta ADADELTA: An Adaptive Learning Rate Method	2012	15
Lookahead Lookahead Optimizer: k steps forward, 1 step back	2019	13
SMA	2000	11
AdaBound Adaptive Gradient Methods with Dynamic Bound of Learning Rate	2019	11
Demon Demon: Improved Neural Network Training with Momentum Decay	2019	10
HGS	2000	9
Gradient Checkpointing Training Deep Nets with Sublinear Memory Cost	2016	9
AdaMax Adam: A Method for Stochastic Optimization	2014	7
1-bit LAMB 1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed	2021	7
Forward gradient Gradients without Backpropagation	2022	7
NT-ASGD Regularizing and Optimizing LSTM Language Models	2017	6
Distributed Shampoo Towards Practical Second Order Optimization for Deep Learning	2021	4
SM3 Memory Efficient Adaptive Optimization	2019	4
NADAM	2015	4
AdaShift AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methods	2018	3
PowerSGD PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization	2019	3
gCANS Adaptive shot allocation for fast convergence in variational quantum algorithms	2021	2
QHM Quasi-hyperbolic momentum and Adam for deep learning	2018	2
Polyak Averaging	1991	2
MPSO Motion-Encoded Particle Swarm Optimization for Moving Target Search Using UAVs	2020	2
AdaSmooth AdaSmooth: An Adaptive Learning Rate Method based on Effective Ratio	2022	2
AMSBound Adaptive Gradient Methods with Dynamic Bound of Learning Rate	2019	1
Demon CM Demon: Improved Neural Network Training with Momentum Decay	2019	1
SLAMB SLAMB: Accelerated Large Batch Training with Sparse Communication	2023	1
SRMM Stochastic regularized majorization-minimization with weakly convex and multi-convex surrogates	2022	1
AdaMod An Adaptive and Momental Bound Method for Stochastic Learning	2019	1
AdaSqrt Second-order Information in First-order Optimization Methods	2019	1
DSPT Adaptive hybrid activation function for deep neural networks	2022	1
Powerpropagation Powerpropagation: A sparsity inducing weight reparameterisation	2021	1
SGDW Decoupled Weight Decay Regularization	2017	1
Demon ADAM Demon: Improved Neural Network Training with Momentum Decay	2019	1
YellowFin YellowFin and the Art of Momentum Tuning	2017	1
AggMo Aggregated Momentum: Stability Through Passive Damping	2018	1
ATMO Combining Optimization Methods Using an Adaptive Meta Optimizer	2021	1
MADGRAD Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization	2021	1
QHAdam Quasi-hyperbolic momentum and Adam for deep learning	2018	1
FASFA	2000	0

Stochastic Optimization Edit

Methods Add a Method

Stochastic Optimization

Edit

Methods

Add a Method