Search Results for author:
Found 7 papers, 3 papers with code
Weight decay induces low-rank attention layers
The effect of regularizers such as weight decay when training deep neural networks is not well understood.
Learning Randomized Algorithms with Transformers
Randomization is a powerful tool that endows algorithms with remarkable properties.
When can transformers compositionally generalize in-context?
Under what circumstances can transformers compositionally generalize from a subset of tasks to all possible combinations of tasks that share similar components?
Attention as a Hypernetwork
To further examine the hypothesis that the intrinsic hypernetwork of multi-head attention supports compositional generalization, we ablate whether making the hypernetwork generated linear value network nonlinear strengthens compositionality.
Discovering modular solutions that generalize compositionally
This allows us to relate the problem of compositional generalization to that of identification of the underlying modules.
Gated recurrent neural networks discover attention
In particular, we examine RNNs trained to solve simple in-context learning tasks on which Transformers are known to excel and find that gradient descent instills in our RNNs the same attention-based in-context learning algorithm used by Transformers.
Random initialisations performing above chance and how to find them
Neural networks trained with stochastic gradient descent (SGD) starting from different random initialisations typically find functionally very similar solutions, raising the question of whether there are meaningful differences between different SGD solutions.
