Learning Rate Schedules

Linear Warmup With Cosine Annealing

Linear Warmup With Cosine Annealing is a learning rate schedule where we increase the learning rate linearly for $n$ updates and then anneal according to a cosine schedule afterwards.

Papers


Paper Code Results Date Stars

Categories