Search Results for author:
Found 21 papers, 8 papers with code
Flexible task abstractions emerge in linear networks with fast and bounded units
We observe that the weights self-organize into modules specialized for tasks or sub-tasks encountered, while the gates layer forms unique representations that switch the appropriate weight modules (task abstractions).
On The Specialization of Neural Modules
Our results shed light on the difficulty of module specialization, what is required for modules to successfully specialize, and the necessity of modular architectures to achieve systematicity.
From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks
Biological and artificial neural networks develop internal representations that enable them to perform complex tasks.
What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation
By clamping subsets of activations throughout training, we then identify three underlying subcircuits that interact to drive IH formation, yielding the phase change.
When Representations Align: Universality in Representation Learning Dynamics
We show through experiments that the effective theory describes aspects of representation learning dynamics across a range of deep networks with different activation functions and architectures, and exhibits phenomena similar to the "rich" and "lazy" regime.
The Transient Nature of Emergent In-Context Learning in Transformers
The transient nature of ICL is observed in transformers across a range of model sizes and datasets, raising the question of how much to "overtrain" transformers when seeking compact, cheaper-to-run models.
Meta-Learning Strategies through Value Maximization in Neural Networks
Understanding how to make these meta-learning choices could offer normative accounts of cognitive control functions in biological learners and improve engineered systems.
Abrupt and spontaneous strategy switches emerge in simple regularised neural networks
Humans sometimes have an insight that leads to a sudden and drastic performance improvement on the task they are working on.
The Neural Race Reduction: Dynamics of Abstraction in Gated Networks
Our theoretical understanding of deep learning has not kept pace with its empirical success.
Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup
Deep neural networks achieve stellar generalisation even when they have enough parameters to easily fit all their training data.
Generalisation dynamics of online learning in over-parameterised neural networks
Deep neural networks achieve stellar generalisation on a variety of problems, despite often being large enough to easily fit all their training data.
A mathematical theory of semantic development in deep neural networks
An extensive body of empirical research has revealed remarkable regularities in the acquisition, organization, deployment, and neural representation of human semantic knowledge, thereby raising a fundamental conceptual question: what are the theoretical principles governing the ability of neural networks to acquire, organize, and deploy abstract knowledge by integrating across many individual experiences?
Minnorm training: an algorithm for training over-parameterized deep neural networks
To solve this constrained optimization problem, our method employs Lagrange multipliers that act as integrators of error over training and identify `support vector'-like examples.
Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learning
We derive a correspondence between parameter inference and free energy minimisation in statistical physics.
High-dimensional dynamics of generalization error in neural networks
We study the practically-relevant "high-dimensional" regime where the number of free parameters in the network is on the order of or even larger than the number of examples in the dataset.
Hierarchy Through Composition with Multitask LMDPs
Hierarchical architectures are critical to the scalability of reinforcement learning methods.
Hierarchical Subtask Discovery With Non-Negative Matrix Factorization
Hierarchical reinforcement learning methods offer a powerful means of planning flexible behavior in complicated domains.
Hierarchy through Composition with Linearly Solvable Markov Decision Processes
Hierarchical architectures are critical to the scalability of reinforcement learning methods.
Active Long Term Memory Networks
Continual Learning in artificial neural networks suffers from interference and forgetting when different tasks are learned sequentially.
Qualitatively characterizing neural network optimization problems
Training neural networks involves solving large-scale non-convex optimization problems.
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
We further exhibit a new class of random orthogonal initial conditions on weights that, like unsupervised pre-training, enjoys depth independent learning times.
