Search Results for author:
Found 17 papers, 13 papers with code
Training Neural Networks from Scratch with Parallel Low-Rank Adapters
The scalability of deep learning models is fundamentally limited by computing resources, memory, and communication.
A Spectral Condition for Feature Learning
The push to train ever larger neural networks has motivated the study of initialization and training at large network width.
SketchOGD: Memory-Efficient Continual Learning
This paper proposes a memory-efficient solution to catastrophic forgetting, improving upon an established algorithm known as orthogonal gradient descent (OGD).
Automatic Gradient Descent: Deep Learning without Hyperparameters
Automatic gradient descent trains both fully-connected and convolutional networks out-of-the-box and at ImageNet scale.
Optimisation & Generalisation in Networks of Neurons
On generalisation, a new correspondence is proposed between ensembles of networks and individual networks.
Investigating Generalization by Controlling Normalized Margin
Weight norm $\|w\|$ and margin $\gamma$ participate in learning theory via the normalized margin $\gamma/\|w\|$.
Kernel Interpolation as a Bayes Point Machine
A Bayes point machine is a single classifier that approximates the majority decision of an ensemble of classifiers.
On the Implicit Biases of Architecture & Gradient Descent
Do neural networks generalise because of bias in the functions returned by gradient descent, or bias already present in the network architecture?
Fine-Grained System Identification of Nonlinear Neural Circuits
We study the problem of sparse nonlinear model recovery of high dimensional compositional functions.
Computing the Information Content of Trained Neural Networks
A simple resolution to this conundrum is that the number of weights is usually a bad proxy for the actual amount of information stored.
Learning by Turning: Neural Architecture Aware Optimisation
To address this problem, this paper conducts a combined study of neural architecture and optimisation, leading to a new optimiser called Nero: the neuronal rotator.
Learning compositional functions via multiplicative weight updates
This paper proves that multiplicative weight updates satisfy a descent lemma tailored to compositional functions.
On the distance between two neural networks and the stability of learning
This paper relates parameter distance to gradient breakdown for a broad class of nonlinear compositional functions.
signSGD with Majority Vote is Communication Efficient And Fault Tolerant
Workers transmit only the sign of their gradient vector to a server, and the overall update is decided by a majority vote.
Stochastic Activation Pruning for Robust Adversarial Defense
Neural networks are known to be vulnerable to adversarial examples.
signSGD: Compressed Optimisation for Non-Convex Problems
Using a theorem by Gauss we prove that majority vote can achieve the same reduction in variance as full precision distributed SGD.
Convergence rate of sign stochastic gradient descent for non-convex functions
The sign stochastic gradient descent method (signSGD) utilizes only the sign of the stochastic gradient in its updates.
