Search Results for author:
Found 9 papers, 5 papers with code
Uncovering mesa-optimization algorithms in Transformers
Transformers have become the dominant model in deep learning, but the reason for their superior performance is poorly understood.
Gated recurrent neural networks discover attention
In particular, we examine RNNs trained to solve simple in-context learning tasks on which Transformers are known to excel and find that gradient descent instills in our RNNs the same attention-based in-context learning algorithm used by Transformers.
Online learning of long-range dependencies
Online learning holds the promise of enabling efficient long-term credit assignment in recurrent neural networks.
Random initialisations performing above chance and how to find them
Neural networks trained with stochastic gradient descent (SGD) starting from different random initialisations typically find functionally very similar solutions, raising the question of whether there are meaningful differences between different SGD solutions.
The least-control principle for local learning at equilibrium
As special cases, they include models of great current interest in both neuroscience and machine learning, such as deep neural networks, equilibrium recurrent neural networks, deep equilibrium models, or meta-learning.
Beyond backpropagation: bilevel optimization through implicit differentiation and equilibrium propagation
This paper reviews gradient-based techniques to solve bilevel optimization problems.
Learning where to learn: Gradient sparsity in meta and continual learning
We find that patterned sparsity emerges from this process, with the pattern of sparsity varying on a problem-by-problem basis.
A contrastive rule for meta-learning
Humans and other animals are capable of improving their learning performance as they solve related tasks from a given problem domain, to the point of being able to learn from extremely limited data.
Learning where to learn
Finding neural network weights that generalize well from small datasets is difficult.
