Search Results for author:
Found 23 papers, 16 papers with code
On How Iterative Magnitude Pruning Discovers Local Receptive Fields in Fully Connected Neural Networks
We develop a new method for measuring the effect of individual weights on the statistics of the FCN representations ("cavity method"), which allows us to find evidence in support of this hypothesis.
A distributional simplicity bias in the learning dynamics of transformers
The remarkable capability of over-parameterised neural networks to generalise effectively has been explained by invoking a ``simplicity bias'': neural networks prevent overfitting by initially learning simple classifiers before progressing to more complex, non-linear functions.
Sliding down the stairs: how correlated latent variables accelerate learning with neural networks
In particular, higher-order input cumulants (HOCs) are crucial for their performance.
Learning from higher-order statistics, efficiently: hypothesis tests, random features, and neural networks
Our results show that neural networks extract information from higher-ordercorrelations in the spiked cumulant model efficiently, and reveal a large gap in the amount of data required by neural networks and random features to learn from higher-order cumulants.
Quantifying lottery tickets under label noise: accuracy, calibration, and complexity
Pruning deep neural networks is a widely used strategy to alleviate the computational burden in machine learning.
The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensions
Reinforcement learning (RL) algorithms have proven transformative in a range of domains.
Attacks on Online Learners: a Teacher-Student Analysis
Machine learning models are famously vulnerable to adversarial attacks: small ad-hoc perturbations of the data that can catastrophically alter the model predictions.
Mapping of attention mechanisms to a generalized Potts model
In MLM, a word is randomly masked in an input sequence, and the network is trained to predict the missing word.
Neural networks trained with SGD learn distributions of increasing complexity
The ability of deep neural networks to generalise well even when they interpolate their training data has been explained using various "simplicity biases".
The impact of memory on learning sequence-to-sequence tasks
Here, we propose a simple model for a seq2seq task that has the advantage of providing explicit control over the degree of memory, or non-Markovianity, in the sequences -- the stochastic switching-Ornstein-Uhlenbeck (SSOU) model.
Maslow's Hammer for Catastrophic Forgetting: Node Re-Use vs Node Activation
Continual learning - learning new tasks in sequence while maintaining performance on old tasks - remains particularly challenging for artificial neural networks.
Data-driven emergence of convolutional structure in neural networks
Here, we show how initially fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs, resulting in localised, space-tiling receptive fields.
The dynamics of representation learning in shallow, non-linear autoencoders
We derive a set of asymptotically exact equations that describe the generalisation dynamics of autoencoders trained with stochastic gradient descent (SGD) in the limit of high-dimensional inputs.
Continual Learning in the Teacher-Student Setup: Impact of Task Similarity
Using each teacher to represent a different task, we investigate how the relationship between teachers affects the amount of forgetting and transfer exhibited by the student when the task switches.
Redundant representations help generalization in wide neural networks
Deep neural networks (DNNs) defy the classical bias-variance trade-off: adding parameters to a DNN that interpolates its training data will typically improve its generalization performance.
Bayesian reconstruction of memories stored in neural networks from their connectivity
The advent of comprehensive synaptic wiring diagrams of large neural circuits has created the field of connectomics and given rise to a number of open research questions.
Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeed
Here, we show theoretically that two-layer neural networks (2LNN) with only a few hidden neurons can beat the performance of kernel learning on a simple Gaussian mixture classification task.
Learning curves of generic features maps for realistic datasets with a teacher-student model
While still solvable in a closed form, this generalization is able to capture the learning curves for a broad range of realistic data sets, thus redeeming the potential of the teacher-student framework.
Align, then memorise: the dynamics of learning with feedback alignment
Direct Feedback Alignment (DFA) is emerging as an efficient and biologically plausible alternative to the ubiquitous backpropagation algorithm for training deep neural networks.
The Gaussian equivalence of generative models for learning with shallow neural networks
Here, we go beyond this simple paradigm by studying the performance of neural networks trained on data drawn from pre-trained generative models.
Modelling the influence of data structure on learning in neural networks: the hidden manifold model
We demonstrate that learning of the hidden manifold model is amenable to an analytical treatment by proving a "Gaussian Equivalence Property" (GEP), and we use the GEP to show how the dynamics of two-layer neural networks trained using one-pass stochastic gradient descent is captured by a set of integro-differential equations that track the performance of the network at all times.
Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup
Deep neural networks achieve stellar generalisation even when they have enough parameters to easily fit all their training data.
Generalisation dynamics of online learning in over-parameterised neural networks
Deep neural networks achieve stellar generalisation on a variety of problems, despite often being large enough to easily fit all their training data.
