Search Results for author: Aitor Lewkowycz

Found 4 papers, 1 papers with code

How to decay your learning rate

no code implementations23 Mar 2021 Aitor Lewkowycz

Complex learning rate schedules have become an integral part of deep learning.

The large learning rate phase of deep learning

1 code implementation1 Jan 2021 Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, Guy Gur-Ari

In the small learning rate phase, training can be understood using the existing theory of infinitely wide neural networks.

On the training dynamics of deep networks with $L_2$ regularization

no code implementations NeurIPS 2020 Aitor Lewkowycz, Guy Gur-Ari

Finally, we show that these empirical relations can be understood theoretically in the context of infinitely wide networks.

Image Classification

The large learning rate phase of deep learning: the catapult mechanism

no code implementations4 Mar 2020 Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, Guy Gur-Ari

In the small learning rate phase, training can be understood using the existing theory of infinitely wide neural networks.

Cannot find the paper you are looking for? You can Submit a new open access paper.