Search Results for author:
Found 5 papers, 3 papers with code
What to expect of hardware metric predictors in NAS
Due to inaccurate predictions, the selected architectures are generally suboptimal, which we quantify as an expected reduction in accuracy and hypervolume.
Using a one dimensional parabolic model of the full-batch loss to estimate learning rates during training
In the experiments conducted, our approach is on par with SGD with Momentum tuned with a piece-wise constant learning rate schedule and often outperforms other line search approaches for Deep Learning across models, datasets, and batch sizes on validation and test accuracy.
Empirically explaining SGD from a line search perspective
Optimization in Deep Learning is mainly guided by vague intuitions and strong assumptions, with a limited understanding how and why these work in practice.
A straightforward line search approach on the expected empirical loss for stochastic deep learning problems
In traditional optimization, line searches are used to determine good step sizes, however, in deep learning, it is too costly to search for good step sizes on the expected empirical loss due to noisy losses.
Parabolic Approximation Line Search for DNNs
The optimal step size is closely related to the shape of the loss in the update step direction.
