Search Results for author:
Found 4 papers, 4 papers with code
Random initialisations performing above chance and how to find them
Neural networks trained with stochastic gradient descent (SGD) starting from different random initialisations typically find functionally very similar solutions, raising the question of whether there are meaningful differences between different SGD solutions.
Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization
Second-order optimizers are thought to hold the potential to speed up neural network training, but due to the enormous size of the curvature matrix, they typically require approximations to be computationally tractable.
Unifying Regularisation Methods for Continual Learning
Moreover, we show that for SI the relation to the Fisher -- and in fact its performance -- is due to a previously unknown bias.
Optimal Kronecker-Sum Approximation of Real Time Recurrent Learning
In contrast, the online training algorithm Real Time Recurrent Learning (RTRL) provides untruncated gradients, with the disadvantage of impractically large computational costs.
