Stochastic Optimization

Edit

General • Optimization • 71 methods

Stochastic Optimization methods are used to optimize neural networks. We typically take a mini-batch of data, hence 'stochastic', and perform a type of gradient descent with this minibatch. Below you can find a continuously updating list of stochastic optimization algorithms.

Subcategories

1 Large Batch Optimization

2 Momentum Rules

Methods

Add a Method

Method	Year	Papers
Adam Adam: A Method for Stochastic Optimization	2014	19803
SGD	1951	1819
Adafactor Adafactor: Adaptive Learning Rates with Sublinear Memory Cost	2018	623
RMSProp	2013	457
LAMB Large Batch Optimization for Deep Learning: Training BERT in 76 minutes	2019	183
AdaGrad	2011	177
AdamW Decoupled Weight Decay Regularization	2017	156
Gravity Gravity Optimizer: a Kinematic Approach on Optimization in Deep Learning	2021	155
Deep Ensembles Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles	2016	147
SGD with Momentum	1999	141
MAS Mixing ADAM and SGD: a Combined Optimization Method	2020	108
FA Random feedback weights support learning in deep neural networks	2014	104
LARS Large Batch Training of Convolutional Networks	2017	68
Local SGD Local SGD Converges Fast and Communicates Little	2018	61
RAdam On the Variance of the Adaptive Learning Rate and Beyond	2019	53
AO	2000	50
AMSGrad On the Convergence of Adam and Beyond	2019	48
DFA Direct Feedback Alignment Provides Learning in Deep Neural Networks	2016	46
Apollo Apollo: An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimization	2020	41
Stochastic Weight Averaging Averaging Weights Leads to Wider Optima and Better Generalization	2018	39
INFO	2000	34
1-bit Adam 1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed	2021	32
Gradient Sparsification Gradient Sparsification for Communication-Efficient Distributed Optimization	2017	31
KP Deep Learning without Weight Transport	2019	29
Nesterov Accelerated Gradient	1983	28
ECO	2000	20
Adabelief AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients	2020	18
SMA	2000	17
Lookahead Lookahead Optimizer: k steps forward, 1 step back	2019	16
AdaDelta ADADELTA: An Adaptive Learning Rate Method	2012	15
PO	2000	14
Demon Demon: Improved Neural Network Training with Momentum Decay	2019	13
Gradient Checkpointing Training Deep Nets with Sublinear Memory Cost	2016	11
AdaBound Adaptive Gradient Methods with Dynamic Bound of Learning Rate	2019	11
Forward gradient Gradients without Backpropagation	2022	10
HGS	2000	10
1-bit LAMB 1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed	2021	9
AdaMax Adam: A Method for Stochastic Optimization	2014	8
NT-ASGD Regularizing and Optimizing LSTM Language Models	2017	6
AdaHessian ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning	2020	6
NADAM	2015	5
SM3 Memory Efficient Adaptive Optimization	2019	5
Distributed Shampoo Towards Practical Second Order Optimization for Deep Learning	2021	4
AdaShift AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methods	2018	4
PowerSGD PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization	2019	3
gCANS Adaptive shot allocation for fast convergence in variational quantum algorithms	2021	2
QHM Quasi-hyperbolic momentum and Adam for deep learning	2018	2
Polyak Averaging	1991	2
MPSO Motion-Encoded Particle Swarm Optimization for Moving Target Search Using UAVs	2020	2
AdaSmooth AdaSmooth: An Adaptive Learning Rate Method based on Effective Ratio	2022	2
AMSBound Adaptive Gradient Methods with Dynamic Bound of Learning Rate	2019	1
Demon CM Demon: Improved Neural Network Training with Momentum Decay	2019	1
SLAMB SLAMB: Accelerated Large Batch Training with Sparse Communication	2023	1
SRMM Stochastic regularized majorization-minimization with weakly convex and multi-convex surrogates	2022	1
AdaMod An Adaptive and Momental Bound Method for Stochastic Learning	2019	1
AdaSqrt Second-order Information in First-order Optimization Methods	2019	1
DSPT Adaptive hybrid activation function for deep neural networks	2022	1
PLO	2000	1
Adam-mini Adam-mini: Use Fewer Learning Rates To Gain More	2024	1
Powerpropagation Powerpropagation: A sparsity inducing weight reparameterisation	2021	1
SGDW Decoupled Weight Decay Regularization	2017	1
Demon ADAM Demon: Improved Neural Network Training with Momentum Decay	2019	1
YellowFin YellowFin and the Art of Momentum Tuning	2017	1
AggMo Aggregated Momentum: Stability Through Passive Damping	2018	1
ATMO Combining Optimization Methods Using an Adaptive Meta Optimizer	2021	1
MADGRAD Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization	2021	1
AdaFisher AdaFisher: Adaptive Second Order Optimization via Fisher Information	2024	1
QHAdam Quasi-hyperbolic momentum and Adam for deep learning	2018	1
FASFA	2000	0
FATA	2000	0

Stochastic Optimization Edit

Methods Add a Method

Stochastic Optimization

Edit

Methods

Add a Method