Optimization

Edit

General • 96 methods

Subcategories

1 Hybrid Optimization

2 Hyperparameter Search

3 Prioritized Sampling

4 Stochastic Optimization

Methods

Add a Method

Method	Year	Papers
Adam Adam: A Method for Stochastic Optimization	2014	17550
SGD	1951	1744
Adafactor Adafactor: Adaptive Learning Rates with Sublinear Memory Cost	2018	581
RMSProp	2013	430
ADMM	2000	353
Random Search	2000	222
LAMB Large Batch Optimization for Deep Learning: Training BERT in 76 minutes	2019	179
AdaGrad	2011	168
AdamW Decoupled Weight Decay Regularization	2017	148
SGD with Momentum	1999	141
Deep Ensembles Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles	2016	140
Gravity Gravity Optimizer: a Kinematic Approach on Optimization in Deep Learning	2021	133
TTUR GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium	2017	131
Gradient Clipping	2000	131
MAS Mixing ADAM and SGD: a Combined Optimization Method	2020	93
FA Random feedback weights support learning in deep neural networks	2014	91
Sharpness-Aware Minimization Sharpness-Aware Minimization for Efficiently Improving Generalization	2020	80
SAGA SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives	2014	70
AMP Regularizing Neural Networks via Adversarial Model Perturbation	2020	66
LARS Large Batch Training of Convolutional Networks	2017	66
SLR Enabling Retrain-free Deep Neural Network Pruning using Surrogate Lagrangian Relaxation	2020	66
Local SGD Local SGD Converges Fast and Communicates Little	2018	59
DAC Dynamic Algorithm Configuration: Foundation of a New Meta-Algorithmic Framework	2020	51
RAdam On the Variance of the Adaptive Learning Rate and Beyond	2019	50
Natural Gradient Descent	1998	49
AMSGrad On the Convergence of Adam and Beyond	2019	47
DFA Direct Feedback Alignment Provides Learning in Deep Neural Networks	2016	41
Apollo Apollo: An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimization	2020	38
Stochastic Weight Averaging Averaging Weights Leads to Wider Optima and Better Generalization	2018	37
Gradient Sparsification Gradient Sparsification for Communication-Efficient Distributed Optimization	2017	30
1-bit Adam 1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed	2021	30
Nesterov Accelerated Gradient	1983	27
KP Deep Learning without Weight Transport	2019	25
Stochastic Gradient Variational Bayes Auto-Encoding Variational Bayes	2013	20
Adabelief AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients	2020	18
Population Based Training Population Based Training of Neural Networks	2017	18
Lion	2000	17
Lookahead Lookahead Optimizer: k steps forward, 1 step back	2019	16
ATSS Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selection	2019	15
SMA	2000	15
AdaDelta ADADELTA: An Adaptive Learning Rate Method	2012	15
PISA Prime Sample Attention in Object Detection	2019	13
Demon Demon: Improved Neural Network Training with Momentum Decay	2019	11
AdaBound Adaptive Gradient Methods with Dynamic Bound of Learning Rate	2019	11
HGS	2000	10
Gradient Checkpointing Training Deep Nets with Sublinear Memory Cost	2016	10
Forward gradient Gradients without Backpropagation	2022	9
1-bit LAMB 1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed	2021	9
AdaMax Adam: A Method for Stochastic Optimization	2014	8
OHEM Training Region-based Object Detectors with Online Hard Example Mining	2016	7
NT-ASGD Regularizing and Optimizing LSTM Language Models	2017	6
GBO	2000	6
IoU-Balanced Sampling Libra R-CNN: Towards Balanced Learning for Object Detection	2019	5
SM3 Memory Efficient Adaptive Optimization	2019	5
AdaHessian ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning	2020	5
Proximity Regularization Federated Optimization in Heterogeneous Networks	2018	5
Distributed Shampoo Towards Practical Second Order Optimization for Deep Learning	2021	4
Harris Hawks optimization (HHO)	2000	4
NADAM	2015	4
AdaShift AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methods	2018	3
Neural adjoint Benchmarking deep inverse models over time, and the neural-adjoint method	2020	3
PowerSGD PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization	2019	3
AdaSmooth AdaSmooth: An Adaptive Learning Rate Method based on Effective Ratio	2022	2
RUN	2000	2
Tree-structured Parzen Estimator Approach (TPE) Hyperopt: A Python Library for Optimizing the Hyperparameters of Machine Learning Algorithms	2013	2
gCANS Adaptive shot allocation for fast convergence in variational quantum algorithms	2021	2
QHM Quasi-hyperbolic momentum and Adam for deep learning	2018	2
Polyak Averaging	1991	2
MPSO Motion-Encoded Particle Swarm Optimization for Moving Target Search Using UAVs	2020	2
AMSBound Adaptive Gradient Methods with Dynamic Bound of Learning Rate	2019	1
Demon CM Demon: Improved Neural Network Training with Momentum Decay	2019	1
Differentiable Hyperparameter Search sharpDARTS: Faster and More Accurate Differentiable Architecture Search	2019	1
SLAMB SLAMB: Accelerated Large Batch Training with Sparse Communication	2023	1
Grammatical evolution + Q-learning Evolutionary learning of interpretable decision trees	2020	1
SRMM Stochastic regularized majorization-minimization with weakly convex and multi-convex surrogates	2022	1
AdaMod An Adaptive and Momental Bound Method for Stochastic Learning	2019	1
AdaSqrt Second-order Information in First-order Optimization Methods	2019	1
DABMD	2020	1
DSPT Adaptive hybrid activation function for deep neural networks	2022	1
SlowMo SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum	2019	1
Powerpropagation Powerpropagation: A sparsity inducing weight reparameterisation	2021	1
SGDW Decoupled Weight Decay Regularization	2017	1
Demon ADAM Demon: Improved Neural Network Training with Momentum Decay	2019	1
MEUZZ MEUZZ: Smart Seed Scheduling for Hybrid Fuzzing	2020	1
DeepZero DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Training	2023	1
YellowFin YellowFin and the Art of Momentum Tuning	2017	1
AggMo Aggregated Momentum: Stability Through Passive Damping	2018	1
ATMO Combining Optimization Methods Using an Adaptive Meta Optimizer	2021	1
MADGRAD Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization	2021	1
QHAdam Quasi-hyperbolic momentum and Adam for deep learning	2018	1
HFPSO	2000	0
FASFA	2000	0

Optimization Edit

Methods Add a Method

Optimization

Edit

Methods

Add a Method