Search Results for author:
Found 10 papers, 8 papers with code
The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention
Linear layers in neural networks (NNs) trained by gradient descent can be expressed as a key-value memory system which stores all training datapoints and the initial weights, and produces outputs using unnormalised dot attention over the entire training experience.
A Modern Self-Referential Weight Matrix That Learns to Modify Itself
The weight matrix (WM) of a neural network (NN) is its program.
Improving Baselines in the Wild
We share our experience with the recently released WILDS benchmark, a collection of ten datasets dedicated to developing models and training strategies which are robust to domain shifts.
The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization
Despite progress across a broad range of applications, Transformers have limited success in systematic generalization.
Learning Adaptive Control Flow in Transformers for Improved Systematic Generalization
Despite successes across a broad range of applications, Transformers have limited capability in systematic generalization.
Adaptive Control Flow in Transformers Improves Systematic Generalization
Despite successes across a broad range of applications, Transformers have limited capability in systematic generalization.
The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers
Our models improve accuracy from 50% to 85% on the PCFG productivity split, and from 35% to 81% on COGS.
Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
Transformers with linearised attention (''linear Transformers'') have demonstrated the practical scalability and effectiveness of outer product-based Fast Weight Programmers (FWPs) from the '90s.
Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks
Neural networks (NNs) whose subnetworks implement reusable functions are expected to offer numerous advantages, including compositionality through efficient recombination of functional building blocks, interpretability, preventing catastrophic interference, etc.
Improving Differentiable Neural Computers Through Memory Masking, De-allocation, and Link Distribution Sharpness Control
The Differentiable Neural Computer (DNC) can learn algorithmic and question answering tasks.
