Learning Rate Schedules

Edit

General • 13 methods

Learning Rate Schedules refer to schedules for the learning rate during the training of neural networks. Below you can find a continuously updating list of learning rate schedules.

Methods

Add a Method

Method	Year	Papers
Linear Warmup With Linear Decay	2000	6258
Cosine Annealing SGDR: Stochastic Gradient Descent with Warm Restarts	2016	3579
Linear Warmup With Cosine Annealing	2000	3431
Inverse Square Root Schedule	2000	655
Exponential Decay	2000	97
Step Decay	2000	71
Slanted Triangular Learning Rates Universal Language Model Fine-tuning for Text Classification	2018	39
Linear Warmup	2000	35
1cycle A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay	2018	13
Polynomial Rate Decay	2000	10
Cosine Power Annealing sharpDARTS: Faster and More Accurate Differentiable Architecture Search	2019	6
Cyclical Learning Rate Policy	2000	5
Log Decay	2000	0