Search Results for author:
Found 14 papers, 7 papers with code
Weight decay induces low-rank attention layers
The effect of regularizers such as weight decay when training deep neural networks is not well understood.
Multi-agent cooperation through learning-aware policy gradients
Self-interested individuals often fail to cooperate, posing a fundamental challenge for multi-agent learning.
Learning Randomized Algorithms with Transformers
Randomization is a powerful tool that endows algorithms with remarkable properties.
When can transformers compositionally generalize in-context?
Under what circumstances can transformers compositionally generalize from a subset of tasks to all possible combinations of tasks that share similar components?
Attention as a Hypernetwork
To further examine the hypothesis that the intrinsic hypernetwork of multi-head attention supports compositional generalization, we ablate whether making the hypernetwork generated linear value network nonlinear strengthens compositionality.
Discovering modular solutions that generalize compositionally
This allows us to relate the problem of compositional generalization to that of identification of the underlying modules.
Uncovering mesa-optimization algorithms in Transformers
Some autoregressive models exhibit in-context learning capabilities: being able to learn as an input sequence is processed, without undergoing any parameter changes, and without being explicitly trained to do so.
Gated recurrent neural networks discover attention
In particular, we examine RNNs trained to solve simple in-context learning tasks on which Transformers are known to excel and find that gradient descent instills in our RNNs the same attention-based in-context learning algorithm used by Transformers.
Disentangling the Predictive Variance of Deep Ensembles through the Neural Tangent Kernel
Identifying unfamiliar inputs, also known as out-of-distribution (OOD) detection, is a crucial property of any decision making process.
Meta-Learning via Classifier(-free) Diffusion Guidance
Our methods repurpose the popular generative image synthesis techniques of natural language guidance and diffusion models to generate neural network weights adapted for tasks.
The least-control principle for local learning at equilibrium
As special cases, they include models of great current interest in both neuroscience and machine learning, such as deep neural networks, equilibrium recurrent neural networks, deep equilibrium models, or meta-learning.
Learning where to learn: Gradient sparsity in meta and continual learning
We find that patterned sparsity emerges from this process, with the pattern of sparsity varying on a problem-by-problem basis.
Posterior Meta-Replay for Continual Learning
We offer a practical deep learning implementation of our framework based on probabilistic task-conditioned hypernetworks, an approach we term posterior meta-replay.
Neural networks with late-phase weights
The largely successful method of training neural networks is to learn their weights using some variant of stochastic gradient descent (SGD).
Ranked #69 on
Image Classification
on CIFAR-100
(using extra training data)
