Large Batch Optimization

Edit

Methods

Add a Method

Method	Year	Papers
Adam Adam: A Method for Stochastic Optimization	2014	17132
Adafactor Adafactor: Adaptive Learning Rates with Sublinear Memory Cost	2018	571
LAMB Large Batch Optimization for Deep Learning: Training BERT in 76 minutes	2019	177
AdaGrad	2011	164
LARS Large Batch Training of Convolutional Networks	2017	66
Nesterov Accelerated Gradient	1983	27
1-bit Adam 1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed	2021	27
1-bit LAMB 1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed	2021	9
SM3 Memory Efficient Adaptive Optimization	2019	5
Distributed Shampoo Towards Practical Second Order Optimization for Deep Learning	2021	4
NADAM	2015	4
SLAMB SLAMB: Accelerated Large Batch Training with Sparse Communication	2023	1