Search Results for author:
Found 7 papers, 6 papers with code
When can transformers compositionally generalize in-context?
Under what circumstances can transformers compositionally generalize from a subset of tasks to all possible combinations of tasks that share similar components?
Attention as a Hypernetwork
To further examine the hypothesis that the intrinsic hypernetwork of multi-head attention supports compositional generalization, we ablate whether making the hypernetwork generated linear value network nonlinear strengthens compositionality.
Discovering modular solutions that generalize compositionally
This allows us to relate the problem of compositional generalization to that of identification of the underlying modules.
Online learning of long-range dependencies
Online learning holds the promise of enabling efficient long-term credit assignment in recurrent neural networks.
Random initialisations performing above chance and how to find them
Neural networks trained with stochastic gradient descent (SGD) starting from different random initialisations typically find functionally very similar solutions, raising the question of whether there are meaningful differences between different SGD solutions.
Learning where to learn: Gradient sparsity in meta and continual learning
We find that patterned sparsity emerges from this process, with the pattern of sparsity varying on a problem-by-problem basis.
A contrastive rule for meta-learning
Humans and other animals are capable of improving their learning performance as they solve related tasks from a given problem domain, to the point of being able to learn from extremely limited data.
