Search Results for author:
Found 18 papers, 13 papers with code
Neural networks with late-phase weights
The largely successful method of training neural networks is to learn their weights using some variant of stochastic gradient descent (SGD).
Ranked #70 on
Image Classification
on CIFAR-100
(using extra training data)
Transformers learn in-context by gradient descent
We start by providing a simple weight construction that shows the equivalence of data transformations induced by 1) a single linear self-attention layer and by 2) gradient-descent (GD) on a regression loss.
Continual learning with hypernetworks
Artificial neural networks suffer from catastrophic forgetting when they are sequentially trained on multiple tasks.
Ranked #4 on
Continual Learning
on F-CelebA (10 tasks)
Posterior Meta-Replay for Continual Learning
We offer a practical deep learning implementation of our framework based on probabilistic task-conditioned hypernetworks, an approach we term posterior meta-replay.
Learning where to learn: Gradient sparsity in meta and continual learning
We find that patterned sparsity emerges from this process, with the pattern of sparsity varying on a problem-by-problem basis.
A Theoretical Framework for Target Propagation
Here, we analyze target propagation (TP), a popular but not yet fully understood alternative to BP, from the standpoint of mathematical optimization.
A contrastive rule for meta-learning
Humans and other animals are capable of improving their learning performance as they solve related tasks from a given problem domain, to the point of being able to learn from extremely limited data.
Credit Assignment in Neural Networks through Deep Feedback Control
The success of deep learning sparked interest in whether the brain learns by using similar techniques for assigning credit to each synaptic weight for its contribution to the network output.
Minimizing Control for Credit Assignment with Strong Feedback
Building upon deep feedback control (DFC), a recently proposed credit assignment method, we combine strong feedback influences on neural activity with gradient-based learning and show that this naturally leads to a novel view on neural network optimization.
Online learning of long-range dependencies
Online learning holds the promise of enabling efficient long-term credit assignment in recurrent neural networks.
Dendritic error backpropagation in deep cortical microcircuits
Animal behaviour depends on learning to associate sensory stimuli with the desired motor command.
The least-control principle for local learning at equilibrium
As special cases, they include models of great current interest in both neuroscience and machine learning, such as deep neural networks, equilibrium recurrent neural networks, deep equilibrium models, or meta-learning.
Discovering modular solutions that generalize compositionally
This allows us to relate the problem of compositional generalization to that of identification of the underlying modules.
Dendritic cortical microcircuits approximate the backpropagation algorithm
Deep learning has seen remarkable developments over the last years, many of them inspired by neuroscience.
Conductance-based dendrites perform Bayes-optimal cue integration
We propose a novel, Bayesian view on the dynamics of conductance-based neurons and synapses which suggests that they are naturally equipped to optimally perform information integration.
Beyond backpropagation: bilevel optimization through implicit differentiation and equilibrium propagation
This paper reviews gradient-based techniques to solve bilevel optimization problems.
Gated recurrent neural networks discover attention
In particular, we examine RNNs trained to solve simple in-context learning tasks on which Transformers are known to excel and find that gradient descent instills in our RNNs the same attention-based in-context learning algorithm used by Transformers.
Uncovering mesa-optimization algorithms in Transformers
Transformers have become the dominant model in deep learning, but the reason for their superior performance is poorly understood.
