Search Results for author: Ross M. Clarke

Found 3 papers, 3 papers with code

Adam through a Second-Order Lens

1 code implementation23 Oct 2023 Ross M. Clarke, Baiyu Su, José Miguel Hernández-Lobato

Research into optimisation for deep learning is characterised by a tension between the computational efficiency of first-order, gradient-based methods (such as SGD and Adam) and the theoretical efficiency of second-order, curvature-based methods (such as quasi-Newton methods and K-FAC).

Computational Efficiency Second-order methods

Series of Hessian-Vector Products for Tractable Saddle-Free Newton Optimisation of Neural Networks

1 code implementation23 Oct 2023 Elre T. Oldewage, Ross M. Clarke, José Miguel Hernández-Lobato

A truncation of this infinite series provides a new optimisation algorithm which is scalable and comparable to other first- and second-order optimisation methods in both runtime and optimisation performance.

Scalable One-Pass Optimisation of High-Dimensional Weight-Update Hyperparameters by Implicit Differentiation

1 code implementation ICLR 2022 Ross M. Clarke, Elre T. Oldewage, José Miguel Hernández-Lobato

Machine learning training methods depend plentifully and intricately on hyperparameters, motivating automated strategies for their optimisation.

Cannot find the paper you are looking for? You can Submit a new open access paper.