Optimization

Edit

General • 111 methods

Subcategories

1 Hybrid Optimization

2 Hyperparameter Search

3 Prioritized Sampling

4 Stochastic Optimization

Methods

Add a Method

Method	Year	Papers
Adam Adam: A Method for Stochastic Optimization	2014	21341
SGD	1951	1880
Adafactor Adafactor: Adaptive Learning Rates with Sublinear Memory Cost	2018	659
RMSProp	2013	485
ADMM	2000	375
Random Search	2000	237
LAMB Large Batch Optimization for Deep Learning: Training BERT in 76 minutes	2019	189
AdaGrad	2011	181
AdamW Decoupled Weight Decay Regularization	2017	173
Gravity Gravity Optimizer: a Kinematic Approach on Optimization in Deep Learning	2021	167
Deep Ensembles Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles	2016	157
Gradient Clipping	2000	147
SGD with Momentum	1999	141
TTUR GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium	2017	133
MAS Mixing ADAM and SGD: a Combined Optimization Method	2020	124
Sharpness-Aware Minimization Sharpness-Aware Minimization for Efficiently Improving Generalization	2020	111
FA Random feedback weights support learning in deep neural networks	2014	108
SLR Enabling Retrain-free Deep Neural Network Pruning using Surrogate Lagrangian Relaxation	2020	79
ADOPT ADOPT: Modified Adam Can Converge with Any $β_2$ with the Optimal Rate	2024	77
AMP Regularizing Neural Networks via Adversarial Model Perturbation	2020	76
SAGA SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives	2014	74
LARS Large Batch Training of Convolutional Networks	2017	70
DAC Dynamic Algorithm Configuration: Foundation of a New Meta-Algorithmic Framework	2020	67
Local SGD Local SGD Converges Fast and Communicates Little	2018	64
Natural Gradient Descent	1998	60
RAdam On the Variance of the Adaptive Learning Rate and Beyond	2019	55
AO	2000	54
AMSGrad On the Convergence of Adam and Beyond	2019	49
DFA Direct Feedback Alignment Provides Learning in Deep Neural Networks	2016	48
Apollo Apollo: An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimization	2020	43
Stochastic Weight Averaging Averaging Weights Leads to Wider Optima and Better Generalization	2018	40
1-bit Adam 1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed	2021	34
INFO	2000	34
Gradient Sparsification Gradient Sparsification for Communication-Efficient Distributed Optimization	2017	34
Nesterov Accelerated Gradient	1983	32
KP Deep Learning without Weight Transport	2019	29
Lion	2000	25
PO	2000	22
ECO	2000	21
Stochastic Gradient Variational Bayes Auto-Encoding Variational Bayes	2013	20
Adabelief AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients	2020	18
Population Based Training Population Based Training of Neural Networks	2017	18
SMA	2000	17
ATSS Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selection	2019	16
Lookahead Lookahead Optimizer: k steps forward, 1 step back	2019	16
AdaDelta ADADELTA: An Adaptive Learning Rate Method	2012	16
Demon Demon: Improved Neural Network Training with Momentum Decay	2019	15
PISA Prime Sample Attention in Object Detection	2019	15
Gradient Checkpointing Training Deep Nets with Sublinear Memory Cost	2016	12
AdaBound Adaptive Gradient Methods with Dynamic Bound of Learning Rate	2019	11
Forward gradient Gradients without Backpropagation	2022	10
RSM 0/1 Deep Neural Networks via Block Coordinate Descent	2022	10
HGS	2000	10
AdaMax Adam: A Method for Stochastic Optimization	2014	9
1-bit LAMB 1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed	2021	9
OHEM Training Region-based Object Detectors with Online Hard Example Mining	2016	7
NADAM	2015	7
GBO	2000	6
AdaHessian ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning	2020	6
SM3 Memory Efficient Adaptive Optimization	2019	6
NT-ASGD Regularizing and Optimizing LSTM Language Models	2017	6
IoU-Balanced Sampling Libra R-CNN: Towards Balanced Learning for Object Detection	2019	6
Harris Hawks optimization (HHO)	2000	5
Proximity Regularization Federated Optimization in Heterogeneous Networks	2018	5
AdaShift AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methods	2018	4
Adan Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models	2022	4
Distributed Shampoo Towards Practical Second Order Optimization for Deep Learning	2021	4
PowerSGD PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization	2019	3
Neural adjoint Benchmarking deep inverse models over time, and the neural-adjoint method	2020	3
Tree-structured Parzen Estimator Approach (TPE) Hyperopt: A Python Library for Optimizing the Hyperparameters of Machine Learning Algorithms	2013	3
Desirability functions Desirability and the birth of incomplete preferences	2015	2
AdaSmooth AdaSmooth: An Adaptive Learning Rate Method based on Effective Ratio	2022	2
Sophia Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training	2023	2
gCANS Adaptive shot allocation for fast convergence in variational quantum algorithms	2021	2
QHM Quasi-hyperbolic momentum and Adam for deep learning	2018	2
Polyak Averaging	1991	2
RUN	2000	2
MPSO Motion-Encoded Particle Swarm Optimization for Moving Target Search Using UAVs	2020	2
QHAdam Quasi-hyperbolic momentum and Adam for deep learning	2018	1
AMSBound Adaptive Gradient Methods with Dynamic Bound of Learning Rate	2019	1
Demon CM Demon: Improved Neural Network Training with Momentum Decay	2019	1
TextGrad TextGrad: Automatic "Differentiation" via Text	2024	1
Differentiable Hyperparameter Search sharpDARTS: Faster and More Accurate Differentiable Architecture Search	2019	1
SLAMB SLAMB: Accelerated Large Batch Training with Sparse Communication	2023	1
AdEMAMix The AdEMAMix Optimizer: Better, Faster, Older	2024	1
Grammatical evolution + Q-learning Evolutionary learning of interpretable decision trees	2020	1
SRMM Stochastic regularized majorization-minimization with weakly convex and multi-convex surrogates	2022	1
FATA	2000	1
AdaMod An Adaptive and Momental Bound Method for Stochastic Learning	2019	1
AdaSqrt Second-order Information in First-order Optimization Methods	2019	1
DABMD	2020	1
DSPT Adaptive hybrid activation function for deep neural networks	2022	1
Adam-mini Adam-mini: Use Fewer Learning Rates To Gain More	2024	1
SlowMo SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum	2019	1
Powerpropagation Powerpropagation: A sparsity inducing weight reparameterisation	2021	1
SGDW Decoupled Weight Decay Regularization	2017	1
Demon ADAM Demon: Improved Neural Network Training with Momentum Decay	2019	1
MEUZZ MEUZZ: Smart Seed Scheduling for Hybrid Fuzzing	2020	1
DeepZero DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Training	2023	1
YellowFin YellowFin and the Art of Momentum Tuning	2017	1
AggMo Aggregated Momentum: Stability Through Passive Damping	2018	1
ATMO Combining Optimization Methods Using an Adaptive Meta Optimizer	2021	1
MADGRAD Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization	2021	1
AdaFisher AdaFisher: Adaptive Second Order Optimization via Fisher Information	2024	1
PLO	2000	0
HFPSO	2000	0
FASFA	2000	0

Optimization Edit

Methods Add a Method

Optimization

Edit

Methods

Add a Method