Search Results for author:
Found 14 papers, 7 papers with code
Powerpropagation: A sparsity inducing weight reparameterisation
The training of sparse neural networks is becoming an increasingly important tool for reducing the computational footprint of models at training and evaluation, as well enabling the effective scaling up of models.
Top-KAST: Top-K Always Sparse Training
Sparse neural networks are becoming increasingly important as the field seeks to improve the performance of existing models by scaling them up, while simultaneously trying to reduce power consumption and computational footprint.
Perception-Prediction-Reaction Agents for Deep Reinforcement Learning
We employ a temporal hierarchy, using a slow-ticking recurrent core to allow information to flow more easily over long time spans, and three fast-ticking recurrent cores with connections designed to create an information asymmetry.
Multiplicative Interactions and Where to Find Them
We explore the role of multiplicative interaction as a unifying framework to describe a range of classical and modern neural network architectural motifs, such as gating, attention layers, hypernetworks, and dynamic convolutions amongst others.
Compressive Transformers for Long-Range Sequence Modelling
We present the Compressive Transformer, an attentive sequence model which compresses past memories for long-range sequence learning.
Ranked #2 on
Language Modelling
on Hutter Prize
Stabilizing Transformers for Reinforcement Learning
Harnessing the transformer's ability to process long time horizons of information could provide a similar performance boost in partially observable reinforcement learning (RL) domains, but the large-scale transformers used in NLP have yet to be successfully applied to the RL setting.
Meta-learning of Sequential Strategies
In this report we review memory-based meta-learning as a tool for building sample-efficient strategies that learn from past experience to adapt to any task within a target class.
Information asymmetry in KL-regularized RL
In this work we study the possibility of leveraging such repeated structure to speed up and regularize learning.
Distilling Policy Distillation
The transfer of knowledge from one policy to another is an important tool in Deep Reinforcement Learning.
Adapting Auxiliary Losses Using Gradient Similarity
One approach to deal with the statistical inefficiency of neural networks is to rely on auxiliary losses that help to build useful representations.
Mix&Match - Agent Curricula for Reinforcement Learning
(2) We further show that M&M can be used successfully to progress through a curriculum of architectural variants defining an agents internal state.
Been There, Done That: Meta-Learning with Episodic Recall
Meta-learning agents excel at rapidly learning new tasks from open-ended task distributions; yet, they forget what they learn about each task as soon as the next begins.
Low-pass Recurrent Neural Networks - A memory architecture for longer-term correlation discovery
Reinforcement learning (RL) agents performing complex tasks must be able to remember observations and actions across sizable time intervals.
Memory-based Parameter Adaptation
Deep neural networks have excelled on a wide range of problems, from vision to language and game playing.
