Large Batch Optimization

Edit

Methods

Add a Method

Method	Year	Papers
Adam Adam: A Method for Stochastic Optimization	2014	23507
Adafactor Adafactor: Adaptive Learning Rates with Sublinear Memory Cost	2018	715
LAMB Large Batch Optimization for Deep Learning: Training BERT in 76 minutes	2019	196
AdaGrad	2011	188
LARS Large Batch Training of Convolutional Networks	2017	75
1-bit Adam 1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed	2021	39
Nesterov Accelerated Gradient	1983	32
1-bit LAMB 1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed	2021	11
SM3 Memory Efficient Adaptive Optimization	2019	7
NADAM	2015	7
Distributed Shampoo Towards Practical Second Order Optimization for Deep Learning	2021	5
SLAMB SLAMB: Accelerated Large Batch Training with Sparse Communication	2023	1