Learning Rate Schedules

Edit

General • 12 methods

Learning Rate Schedules refer to schedules for the learning rate during the training of neural networks. Below you can find a continuously updating list of learning rate schedules.

Methods

Add a Method

Method	Year	Papers
Linear Warmup With Linear Decay	2000	4786
Cosine Annealing SGDR: Stochastic Gradient Descent with Warm Restarts	2016	2097
Linear Warmup With Cosine Annealing	2000	1958
Inverse Square Root Schedule	2000	487
Exponential Decay	2000	72
Step Decay	2000	70
Slanted Triangular Learning Rates Universal Language Model Fine-tuning for Text Classification	2018	35
Linear Warmup	2000	28
1cycle A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay	2018	12
Polynomial Rate Decay	2000	10
Cosine Power Annealing sharpDARTS: Faster and More Accurate Differentiable Architecture Search	2019	6
Cyclical Learning Rate Policy	2000	5