Search Results for author: Simon Osindero

Found 46 papers, 19 papers with code

Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution

1 code implementation28 Sep 2023 Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, Tim Rocktäschel

Driven by an LLM, Promptbreeder mutates a population of task-prompts, and subsequently evaluates them for fitness on a training set.

Scaling Language Models: Methods, Analysis & Insights from Training Gopher

2 code implementations NA 2021 Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent SIfre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d'Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, Geoffrey Irving

Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world.

Abstract Algebra Anachronisms +133

Meta-Learning with Latent Embedding Optimization

5 code implementations ICLR 2019 Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, Raia Hadsell

We show that it is possible to bypass these limitations by learning a data-dependent latent generative representation of model parameters, and performing gradient-based meta-learning in this low-dimensional latent space.

Few-Shot Learning

Conditional Generative Adversarial Nets

63 code implementations6 Nov 2014 Mehdi Mirza, Simon Osindero

Generative Adversarial Nets [8] were recently introduced as a novel way to train generative models.

Descriptive Human action generation

A fast learning algorithm for deep belief nets

1 code implementation Neural Computation 2006 Geoffrey E. Hinton, Simon Osindero, Yee-Whye Teh

Using complementary priors, we derive a fast, greedy algorithm that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory.

Generative Art Using Neural Visual Grammars and Dual Encoders

1 code implementation1 May 2021 Chrisantha Fernando, S. M. Ali Eslami, Jean-Baptiste Alayrac, Piotr Mirowski, Dylan Banarse, Simon Osindero

Whilst there are perhaps only a few scientific methods, there seem to be almost as many artistic methods as there are artists.

CLIP-CLOP: CLIP-Guided Collage and Photomontage

1 code implementation6 May 2022 Piotr Mirowski, Dylan Banarse, Mateusz Malinowski, Simon Osindero, Chrisantha Fernando

The unabated mystique of large-scale neural networks, such as the CLIP dual image-and-text encoder, popularized automatically generated art.

Prompt Engineering

Decoupled Neural Interfaces using Synthetic Gradients

5 code implementations ICML 2017 Max Jaderberg, Wojciech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, Koray Kavukcuoglu

Training directed neural networks typically requires forward-propagating data through a computation graph, followed by backpropagating error signal, to produce weight updates.

Population Based Training of Neural Networks

9 code implementations27 Nov 2017 Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, Chrisantha Fernando, Koray Kavukcuoglu

Neural networks dominate the modern machine learning landscape, but their training and success still suffer from sensitivity to empirical choices of hyperparameters such as model architecture, loss function, and optimisation algorithm.

Machine Translation Model Selection

Cross-dimensional Weighting for Aggregated Deep Convolutional Features

1 code implementation13 Dec 2015 Yannis Kalantidis, Clayton Mellina, Simon Osindero

We propose a simple and straightforward way of creating powerful image representations via cross-dimensional weighting and aggregation of deep convolutional neural network layer outputs.

Image Retrieval

Top-KAST: Top-K Always Sparse Training

2 code implementations NeurIPS 2020 Siddhant M. Jayakumar, Razvan Pascanu, Jack W. Rae, Simon Osindero, Erich Elsen

Sparse neural networks are becoming increasingly important as the field seeks to improve the performance of existing models by scaling them up, while simultaneously trying to reduce power consumption and computational footprint.

Language Modelling

AlgebraNets

1 code implementation12 Jun 2020 Jordan Hoffmann, Simon Schmitt, Simon Osindero, Karen Simonyan, Erich Elsen

Neural networks have historically been built layerwise from the set of functions in ${f: \mathbb{R}^n \to \mathbb{R}^m }$, i. e. with activations and weights/parameters represented by real numbers, $\mathbb{R}$.

Computational Efficiency Image Classification +1

Understanding Synthetic Gradients and Decoupled Neural Interfaces

1 code implementation ICML 2017 Wojciech Marian Czarnecki, Grzegorz Świrszcz, Max Jaderberg, Simon Osindero, Oriol Vinyals, Koray Kavukcuoglu

When training neural networks, the use of Synthetic Gradients (SG) allows layers or modules to be trained without update locking - without waiting for a true error gradient to be backpropagated - resulting in Decoupled Neural Interfaces (DNIs).

Massively Parallel Video Networks

no code implementations ECCV 2018 Joao Carreira, Viorica Patraucean, Laurent Mazare, Andrew Zisserman, Simon Osindero

We introduce a class of causal video understanding models that aims to improve efficiency of video processing by maximising throughput, minimising latency, and reducing the number of clock cycles.

Action Recognition Temporal Action Localization +1

Kickstarting Deep Reinforcement Learning

no code implementations10 Mar 2018 Simon Schmitt, Jonathan J. Hudson, Augustin Zidek, Simon Osindero, Carl Doersch, Wojciech M. Czarnecki, Joel Z. Leibo, Heinrich Kuttler, Andrew Zisserman, Karen Simonyan, S. M. Ali Eslami

Our method places no constraints on the architecture of the teacher or student agents, and it regulates itself to allow the students to surpass their teachers in performance.

reinforcement-learning Reinforcement Learning (RL)

Beautiful and damned. Combined effect of content quality and social ties on user engagement

no code implementations1 Nov 2017 Luca M. Aiello, Rossano Schifanella, Miriam Redi, Stacey Svetlichnaya, Frank Liu, Simon Osindero

Exposure to beauty is double-edged: following people who produce high-quality content increases one's probability of uploading better photos; however, an excessive imbalance between the quality generated by a user and the user's neighbors leads to a decline in engagement.

Recommendation Systems

Sobolev Training for Neural Networks

no code implementations NeurIPS 2017 Wojciech Marian Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Świrszcz, Razvan Pascanu

In many cases we only have access to input-output pairs from the ground truth, however it is becoming more common to have access to derivatives of the target output with respect to the input - for example when the ground truth function is itself a neural network such as in network compression or distillation.

Strategic Attentive Writer for Learning Macro-Actions

no code implementations NeurIPS 2016 Alexander, Vezhnevets, Volodymyr Mnih, John Agapiou, Simon Osindero, Alex Graves, Oriol Vinyals, Koray Kavukcuoglu

We present a novel deep recurrent neural network architecture that learns to build implicit plans in an end-to-end manner by purely interacting with an environment in reinforcement learning setting.

Atari Games

Recursive Recurrent Nets with Attention Modeling for OCR in the Wild

no code implementations CVPR 2016 Chen-Yu Lee, Simon Osindero

We present recursive recurrent neural networks with attention modeling (R$^2$AM) for lexicon-free optical character recognition in natural scene images.

Language Modelling Optical Character Recognition +1

Meta-Learning by the Baldwin Effect

no code implementations6 Jun 2018 Chrisantha Thomas Fernando, Jakub Sygnowski, Simon Osindero, Jane Wang, Tom Schaul, Denis Teplyashin, Pablo Sprechmann, Alexander Pritzel, Andrei A. Rusu

The scope of the Baldwin effect was recently called into question by two papers that closely examined the seminal work of Hinton and Nowlan.

Meta-Learning

Modeling image patches with a directed hierarchy of Markov random fields

no code implementations NeurIPS 2007 Simon Osindero, Geoffrey E. Hinton

We describe an efficient learning procedure for multilayer generative models that combine the best aspects of Markov random fields and deep, directed belief nets.

Mix & Match - Agent Curricula for Reinforcement Learning

no code implementations ICML 2018 Wojciech Czarnecki, Siddhant Jayakumar, Max Jaderberg, Leonard Hasenclever, Yee Whye Teh, Nicolas Heess, Simon Osindero, Razvan Pascanu

We introduce Mix and match (M&M) – a training framework designed to facilitate rapid and effective learning in RL agents that would be too slow or too challenging to train otherwise. The key innovation is a procedure that allows us to automatically form a curriculum over agents.

reinforcement-learning Reinforcement Learning (RL)

Distilling Policy Distillation

no code implementations6 Feb 2019 Wojciech Marian Czarnecki, Razvan Pascanu, Simon Osindero, Siddhant M. Jayakumar, Grzegorz Swirszcz, Max Jaderberg

The transfer of knowledge from one policy to another is an important tool in Deep Reinforcement Learning.

Meta-Learning Deep Energy-Based Memory Models

no code implementations ICLR 2020 Sergey Bartunov, Jack W. Rae, Simon Osindero, Timothy P. Lillicrap

We study the problem of learning associative memory -- a system which is able to retrieve a remembered pattern based on its distorted or incomplete version.

Meta-Learning Retrieval

A Deep Neural Network's Loss Surface Contains Every Low-dimensional Pattern

no code implementations16 Dec 2019 Wojciech Marian Czarnecki, Simon Osindero, Razvan Pascanu, Max Jaderberg

The work "Loss Landscape Sightseeing with Multi-Point Optimization" (Skorokhodov and Burtsev, 2019) demonstrated that one can empirically find arbitrary 2D binary patterns inside loss surfaces of popular neural networks.

Adapting Behaviour for Learning Progress

no code implementations14 Dec 2019 Tom Schaul, Diana Borsa, David Ding, David Szepesvari, Georg Ostrovski, Will Dabney, Simon Osindero

Determining what experience to generate to best facilitate learning (i. e. exploration) is one of the distinguishing features and open challenges in reinforcement learning.

Atari Games

A Practical Sparse Approximation for Real Time Recurrent Learning

no code implementations12 Jun 2020 Jacob Menick, Erich Elsen, Utku Evci, Simon Osindero, Karen Simonyan, Alex Graves

Current methods for training recurrent neural networks are based on backpropagation through time, which requires storing a complete history of network states, and prohibits updating the weights `online' (after every timestep).

Multiplicative Interactions and Where to Find Them

no code implementations ICLR 2020 Siddhant M. Jayakumar, Wojciech M. Czarnecki, Jacob Menick, Jonathan Schwarz, Jack Rae, Simon Osindero, Yee Whye Teh, Tim Harley, Razvan Pascanu

We explore the role of multiplicative interaction as a unifying framework to describe a range of classical and modern neural network architectural motifs, such as gating, attention layers, hypernetworks, and dynamic convolutions amongst others.

Inductive Bias

Small Data, Big Decisions: Model Selection in the Small-Data Regime

no code implementations ICML 2020 Jorg Bornschein, Francesco Visin, Simon Osindero

Highly overparametrized neural networks can display curiously strong generalization performance - a phenomenon that has recently garnered a wealth of theoretical and empirical research in order to better understand it.

Model Selection

Practical Real Time Recurrent Learning with a Sparse Approximation

no code implementations ICLR 2021 Jacob Menick, Erich Elsen, Utku Evci, Simon Osindero, Karen Simonyan, Alex Graves

For highly sparse networks, SnAp with $n=2$ remains tractable and can outperform backpropagation through time in terms of learning speed when updates are done online.

From Language Games to Drawing Games

no code implementations6 Oct 2020 Chrisantha Fernando, Daria Zenkova, Stanislav Nikolov, Simon Osindero

We attempt to automate various artistic processes by inventing a set of drawing games, analogous to the approach taken by emergent language research in inventing communication games.

Contrastive Topographic Models: Energy-based density models applied to the understanding of sensory coding and cortical topography

no code implementations5 Nov 2020 Simon Osindero

We address the problem of building theoretical models that help elucidate the function of the visual brain at computational/algorithmic and structural/mechanistic levels.

Density Estimation

Entropic Desired Dynamics for Intrinsic Control

no code implementations NeurIPS 2021 Steven Hansen, Guillaume Desjardins, Kate Baumli, David Warde-Farley, Nicolas Heess, Simon Osindero, Volodymyr Mnih

An agent might be said, informally, to have mastery of its environment when it has maximised the effective number of states it can reliably reach.

Montezuma's Revenge

Model-Value Inconsistency as a Signal for Epistemic Uncertainty

no code implementations8 Dec 2021 Angelos Filos, Eszter Vértes, Zita Marinho, Gregory Farquhar, Diana Borsa, Abram Friesen, Feryal Behbahani, Tom Schaul, André Barreto, Simon Osindero

Unlike prior work which estimates uncertainty by training an ensemble of many models and/or value functions, this approach requires only the single model and value function which are already being learned in most model-based reinforcement learning algorithms.

Model-based Reinforcement Learning Rolling Shutter Correction

Solving MaxSAT with Matrix Multiplication

no code implementations1 Nov 2023 David Warde-Farley, Vinod Nair, Yujia Li, Ivan Lobov, Felix Gimeno, Simon Osindero

Since matrix multiplication is the main computational primitive for block Gibbs sampling in an RBM, our approach leads to an elegantly simple algorithm (40 lines of JAX) well-suited for neural network accelerators.

Cannot find the paper you are looking for? You can Submit a new open access paper.