Search Results for author:
Found 4 papers, 3 papers with code
Mixture-of-Depths: Dynamically allocating compute in transformer-based language models
Our method enforces a total compute budget by capping the number of tokens ($k$) that can participate in the self-attention and MLP computations at a given layer.
How to Learn and Represent Abstractions: An Investigation using Symbolic Alchemy
Alchemy is a new meta-learning environment rich enough to contain interesting abstractions, yet simple enough to make fine-grained analysis tractable.
Synthetic Returns for Long-Term Credit Assignment
We propose state-associative (SA) learning, where the agent learns associations between states and arbitrarily distant future rewards, then propagates credit directly between the two.
Rapid Task-Solving in Novel Environments
We show that EPNs learn to execute a value iteration-like planning algorithm and that they generalize to situations beyond their training experience.
