Search Results for author:
Found 23 papers, 7 papers with code
Revisiting the Role of Relearning in Semantic Dementia
While relearning of lost knowledge has been shown in acute brain injuries such as stroke, it has not been widely supported in chronic cognitive diseases such as SD.
Training Dynamics of In-Context Learning in Linear Attention
For the merged parametrization, we show the training dynamics has two fixed points and the loss trajectory exhibits a single, abrupt drop.
Early learning of the optimal constant solution in neural networks and humans
We identify hallmarks of this early OCS phase and illustrate how these signatures are observed in deep linear networks and larger, more complex (and nonlinear) convolutional neural networks solving a hierarchical learning task based on MNIST and CIFAR10.
When Are Bias-Free ReLU Networks Effectively Linear Networks?
We then show that, under symmetry conditions on the data, these networks have the same learning dynamics as linear networks.
Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning
While the impressive performance of modern neural networks is often attributed to their capacity to efficiently extract task-relevant features from data, the mechanisms underlying this rich feature learning regime remain elusive, with much of our theoretical understanding stemming from the opposing lazy regime.
Tilting the Odds at the Lottery: the Interplay of Overparameterisation and Curricula in Neural Networks
A wide range of empirical and theoretical works have shown that overparameterisation can amplify the performance of neural networks.
Why Do Animals Need Shaping? A Theory of Task Composition and Curriculum Learning
Diverse studies in systems neuroscience begin with extended periods of curriculum training known as `shaping' procedures.
Understanding Unimodal Bias in Multimodal Deep Linear Networks
This is the first work to calculate the duration of the unimodal phase in learning as a function of the depth at which modalities are fused within the network, dataset statistics, and initialization.
Are task representations gated in macaque prefrontal cortex?
A recent paper (Flesch et al, 2022) describes behavioural and neural data suggesting that task representations are gated in the prefrontal cortex in both humans and macaques.
The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensions
Reinforcement learning (RL) algorithms have proven transformative in a range of domains.
Continual task learning in natural and artificial agents
How do humans and other animals learn new tasks?
Know your audience: specializing grounded language models with listener subtraction
Through controlled experiments, we show that training a speaker with two listeners that perceive differently, using our method, allows the speaker to adapt to the idiosyncracies of the listeners.
Maslow's Hammer for Catastrophic Forgetting: Node Re-Use vs Node Activation
Continual learning - learning new tasks in sequence while maintaining performance on old tasks - remains particularly challenging for artificial neural networks.
Modelling continual learning in humans with Hebbian context gating and exponentially decaying task signals
Here, we propose novel computational constraints for artificial neural networks, inspired by earlier work on gating in the primate prefrontal cortex, that capture the cost of interleaved training and allow the network to learn two tasks in sequence without forgetting.
Continual Learning in the Teacher-Student Setup: Impact of Task Similarity
Using each teacher to represent a different task, we investigate how the relationship between teachers affects the amount of forgetting and transfer exhibited by the student when the task switches.
An Analytical Theory of Curriculum Learning in Teacher-Student Networks
To study the former, we provide an exact description of the online learning setting, confirming the long-standing experimental observation that curricula can modestly speed up learning.
Probing transfer learning with a model of synthetic correlated datasets
Transfer learning can significantly improve the sample efficiency of neural networks, by exploiting the relatedness between a data-scarce target task and a data-abundant source task.
Characterizing emergent representations in a space of candidate learning rules for deep networks
Studies suggesting that representations in deep networks resemble those in biological brains have mostly relied on one specific learning rule: gradient descent, the workhorse behind modern deep learning.
If deep learning is the answer, then what is the question?
In this Perspective, our goal is to offer a roadmap for systems neuroscience research in the age of deep learning.
Neurons and Cognition
Are Efficient Deep Representations Learnable?
Specifically, we train deep neural networks to learn two simple functions with known efficient solutions: the parity function and the fast Fourier transform.
Tensor Switching Networks
We present a novel neural network algorithm, the Tensor Switching (TS) network, which generalizes the Rectified Linear Unit (ReLU) nonlinearity to tensor-valued hidden units.
Unsupervised learning models of primary cortical receptive fields and receptive field plasticity
In this work we focus on that component of adaptation which occurs during an organism's lifetime, and show that a number of unsupervised feature learning algorithms can account for features of normal receptive field properties across multiple primary sensory cortices.
Measuring Invariances in Deep Networks
Our evaluation metrics can also be used to evaluate future work in unsupervised deep learning, and thus help the development of future algorithms.
