Search Results for author:
Found 47 papers, 19 papers with code
Neural Compression of Atmospheric States
Atmospheric states derived from reanalysis comprise a substantial portion of weather and climate simulation outputs.
Genie: Generative Interactive Environments
We introduce Genie, the first generative interactive environment trained in an unsupervised manner from unlabelled Internet videos.
Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordia
Agent-based modeling has been around for decades, and applied widely across the social and natural sciences.
Solving MaxSAT with Matrix Multiplication
Since matrix multiplication is the main computational primitive for block Gibbs sampling in an RBM, our approach leads to an elegantly simple algorithm (40 lines of JAX) well-suited for neural network accelerators.
Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution
Driven by an LLM, Promptbreeder mutates a population of task-prompts, and subsequently evaluates them for fitness on a training set.
Perception Test: A Diagnostic Benchmark for Multimodal Video Models
We propose a novel multimodal video benchmark - the Perception Test - to evaluate the perception and reasoning skills of pre-trained multimodal models (e. g. Flamingo, SeViLA, or GPT-4).
Ranked #1 on
Point Tracking
on Perception Test
Perception Test: A Diagnostic Benchmark for Multimodal Models
We propose a novel multimodal benchmark – the Perception Test – that aims to extensively evaluate perception and reasoning skills of multimodal models.
CLIP-CLOP: CLIP-Guided Collage and Photomontage
The unabated mystique of large-scale neural networks, such as the CLIP dual image-and-text encoder, popularized automatically generated art.
Training Compute-Optimal Large Language Models
We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget.
Retrieval-Augmented Reinforcement Learning
In this paper we explore an alternative paradigm in which we train a network to map a dataset of past experiences to optimal behavior.
Unified Scaling Laws for Routed Language Models
The performance of a language model has been shown to be effectively modeled as a power-law in its parameter count.
Improving language models by retrieving from trillions of tokens
We enhance auto-regressive language models by conditioning on document chunks retrieved from a large corpus, based on local similarity with preceding tokens.
Ranked #1 on
Language Modelling
on WikiText-103
(using extra training data)
Model-Value Inconsistency as a Signal for Epistemic Uncertainty
Unlike prior work which estimates uncertainty by training an ensemble of many models and/or value functions, this approach requires only the single model and value function which are already being learned in most model-based reinforcement learning algorithms.
Model-based Reinforcement Learning
Rolling Shutter Correction
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world.
Ranked #1 on
Language Modelling
on StackExchange
Entropic Desired Dynamics for Intrinsic Control
An agent might be said, informally, to have mastery of its environment when it has maximised the effective number of states it can reliably reach.
Top-KAST: Top-K Always Sparse Training
Sparse neural networks are becoming increasingly important as the field seeks to improve the performance of existing models by scaling them up, while simultaneously trying to reduce power consumption and computational footprint.
Generative Art Using Neural Visual Grammars and Dual Encoders
Whilst there are perhaps only a few scientific methods, there seem to be almost as many artistic methods as there are artists.
Practical Real Time Recurrent Learning with a Sparse Approximation
For highly sparse networks, SnAp with $n=2$ remains tractable and can outperform backpropagation through time in terms of learning speed when updates are done online.
Contrastive Topographic Models: Energy-based density models applied to the understanding of sensory coding and cortical topography
We address the problem of building theoretical models that help elucidate the function of the visual brain at computational/algorithmic and structural/mechanistic levels.
From Language Games to Drawing Games
We attempt to automate various artistic processes by inventing a set of drawing games, analogous to the approach taken by emergent language research in inventing communication games.
Small Data, Big Decisions: Model Selection in the Small-Data Regime
Highly overparametrized neural networks can display curiously strong generalization performance - a phenomenon that has recently garnered a wealth of theoretical and empirical research in order to better understand it.
AlgebraNets
Neural networks have historically been built layerwise from the set of functions in ${f: \mathbb{R}^n \to \mathbb{R}^m }$, i. e. with activations and weights/parameters represented by real numbers, $\mathbb{R}$.
A Practical Sparse Approximation for Real Time Recurrent Learning
Current methods for training recurrent neural networks are based on backpropagation through time, which requires storing a complete history of network states, and prohibits updating the weights `online' (after every timestep).
Multiplicative Interactions and Where to Find Them
We explore the role of multiplicative interaction as a unifying framework to describe a range of classical and modern neural network architectural motifs, such as gating, attention layers, hypernetworks, and dynamic convolutions amongst others.
A Deep Neural Network's Loss Surface Contains Every Low-dimensional Pattern
The work "Loss Landscape Sightseeing with Multi-Point Optimization" (Skorokhodov and Burtsev, 2019) demonstrated that one can empirically find arbitrary 2D binary patterns inside loss surfaces of popular neural networks.
Adapting Behaviour for Learning Progress
Determining what experience to generate to best facilitate learning (i. e. exploration) is one of the distinguishing features and open challenges in reinforcement learning.
Meta-Learning Deep Energy-Based Memory Models
We study the problem of learning associative memory -- a system which is able to retrieve a remembered pattern based on its distorted or incomplete version.
Meta-learning of Sequential Strategies
In this report we review memory-based meta-learning as a tool for building sample-efficient strategies that learn from past experience to adapt to any task within a target class.
Distilling Policy Distillation
The transfer of knowledge from one policy to another is an important tool in Deep Reinforcement Learning.
Meta-Learning with Latent Embedding Optimization
We show that it is possible to bypass these limitations by learning a data-dependent latent generative representation of model parameters, and performing gradient-based meta-learning in this low-dimensional latent space.
Mix & Match - Agent Curricula for Reinforcement Learning
We introduce Mix and match (M&M) – a training framework designed to facilitate rapid and effective learning in RL agents that would be too slow or too challenging to train otherwise. The key innovation is a procedure that allows us to automatically form a curriculum over agents.
Massively Parallel Video Networks
We introduce a class of causal video understanding models that aims to improve efficiency of video processing by maximising throughput, minimising latency, and reducing the number of clock cycles.
Meta-Learning by the Baldwin Effect
The scope of the Baldwin effect was recently called into question by two papers that closely examined the seminal work of Hinton and Nowlan.
Mix&Match - Agent Curricula for Reinforcement Learning
(2) We further show that M&M can be used successfully to progress through a curriculum of architectural variants defining an agents internal state.
Kickstarting Deep Reinforcement Learning
Our method places no constraints on the architecture of the teacher or student agents, and it regulates itself to allow the students to surpass their teachers in performance.
Population Based Training of Neural Networks
Neural networks dominate the modern machine learning landscape, but their training and success still suffer from sensitivity to empirical choices of hyperparameters such as model architecture, loss function, and optimisation algorithm.
Beautiful and damned. Combined effect of content quality and social ties on user engagement
Exposure to beauty is double-edged: following people who produce high-quality content increases one's probability of uploading better photos; however, an excessive imbalance between the quality generated by a user and the user's neighbors leads to a decline in engagement.
Sobolev Training for Neural Networks
In many cases we only have access to input-output pairs from the ground truth, however it is becoming more common to have access to derivatives of the target output with respect to the input - for example when the ground truth function is itself a neural network such as in network compression or distillation.
FeUdal Networks for Hierarchical Reinforcement Learning
We introduce FeUdal Networks (FuNs): a novel architecture for hierarchical reinforcement learning.
Hierarchical Reinforcement Learning
reinforcement-learning
+2
Understanding Synthetic Gradients and Decoupled Neural Interfaces
When training neural networks, the use of Synthetic Gradients (SG) allows layers or modules to be trained without update locking - without waiting for a true error gradient to be backpropagated - resulting in Decoupled Neural Interfaces (DNIs).
Decoupled Neural Interfaces using Synthetic Gradients
Training directed neural networks typically requires forward-propagating data through a computation graph, followed by backpropagating error signal, to produce weight updates.
Strategic Attentive Writer for Learning Macro-Actions
We present a novel deep recurrent neural network architecture that learns to build implicit plans in an end-to-end manner by purely interacting with an environment in reinforcement learning setting.
Recursive Recurrent Nets with Attention Modeling for OCR in the Wild
We present recursive recurrent neural networks with attention modeling (R$^2$AM) for lexicon-free optical character recognition in natural scene images.
Cross-dimensional Weighting for Aggregated Deep Convolutional Features
We propose a simple and straightforward way of creating powerful image representations via cross-dimensional weighting and aggregation of deep convolutional neural network layer outputs.
Ranked #13 on
Image Retrieval
on RParis (Medium)
Conditional Generative Adversarial Nets
Generative Adversarial Nets [8] were recently introduced as a novel way to train generative models.
Ranked #2 on
Human action generation
on NTU RGB+D 120
Modeling image patches with a directed hierarchy of Markov random fields
We describe an efficient learning procedure for multilayer generative models that combine the best aspects of Markov random fields and deep, directed belief nets.
A fast learning algorithm for deep belief nets
Using complementary priors, we derive a fast, greedy algorithm that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory.
