Search Results for author:
Found 111 papers, 68 papers with code
Optimizing Memory Mapping Using Deep Reinforcement Learning
We also introduce a Reinforcement Learning agent, mallocMuZero, and show that it is capable of playing this game to discover new and improved memory mapping solutions that lead to faster execution times on real ML workloads on ML accelerators.
GraphCast: Learning skillful medium-range global weather forecasting
We introduce a machine-learning (ML)-based weather simulator--called "GraphCast"--which outperforms the most accurate deterministic operational medium-range weather forecasting system in the world, as well as all previous ML baselines.
Emergent Abilities of Large Language Models
Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks.
A Generalist Agent
Inspired by progress in large-scale language modeling, we apply a similar approach towards building a single generalist agent beyond the realm of text outputs.
Ranked #1 on
Skill Generalization
on RGB-Stacking
Flamingo: a Visual Language Model for Few-Shot Learning
Building models that can be rapidly adapted to novel tasks using only a handful of annotated examples is an open challenge for multimodal machine learning research.
Ranked #1 on
Temporal/Casual QA
on NExT-QA
Training Compute-Optimal Large Language Models
We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget.
Integrating Language Guidance into Vision-based Deep Metric Learning
This causes learned embedding spaces to encode incomplete semantic context and misrepresent the semantic relation between classes, impacting the generalizability of the learned metric space.
Ranked #6 on
Metric Learning
on CARS196
(using extra training data)
Non-isotropy Regularization for Proxy-based Deep Metric Learning
Deep Metric Learning (DML) aims to learn representation spaces on which semantic relations can simply be expressed through predefined distance metrics.
Ranked #9 on
Metric Learning
on CUB-200-2011
(using extra training data)
HiP: Hierarchical Perceiver
This however hinders them from scaling up to the inputs sizes required to process raw high-resolution images or video.
General-purpose, long-context autoregressive modeling with Perceiver AR
Real-world data is high-dimensional: a book, image, or musical performance can easily contain hundreds of thousands of elements even after compression.
MuZero with Self-competition for Rate Control in VP9 Video Compression
Specifically, we target the problem of learning a rate control policy to select the quantization parameters (QP) in the encoding process of libvpx, an open source VP9 video compression library widely used by popular video-on-demand (VOD) services.
Competition-Level Code Generation with AlphaCode
Programming is a powerful and ubiquitous problem-solving tool.
Ranked #1 on
Code Generation
on CodeContests
Unified Scaling Laws for Routed Language Models
The performance of a language model has been shown to be effectively modeled as a power-law in its parameter count.
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world.
Ranked #1 on
Abstract Algebra
on BIG-bench
Improving language models by retrieving from trillions of tokens
We enhance auto-regressive language models by conditioning on document chunks retrieved from a large corpus, based on local similarity with preceding tokens.
Ranked #9 on
Question Answering
on Natural Questions
Perceiver IO: A General Architecture for Structured Inputs & Outputs
A central goal of machine learning is the development of systems that can solve many problems in as many data domains as possible.
Ranked #1 on
Optical Flow Estimation
on KITTI 2015
(Average End-Point Error metric)
WikiGraphs: A Wikipedia Text - Knowledge Graph Paired Dataset
We present a new dataset of Wikipedia articles each paired with a knowledge graph, to facilitate the research in conditional text generation, graph generation and graph representation learning.
Ranked #1 on
KG-to-Text Generation
on WikiGraphs
Highly accurate protein structure prediction with AlphaFold
Accurate computational approaches are needed to address this gap and to enable large-scale structural bioinformatics.
The Benchmark Lottery
The world of empirical machine learning (ML) strongly relies on benchmarks in order to determine the relative effectiveness of different algorithms and methods.
Multimodal Few-Shot Learning with Frozen Language Models
When trained at sufficient scale, auto-regressive language models exhibit the notable ability to learn a new language task after being prompted with just a few examples.
Ranked #14 on
Visual Question Answering (VQA)
on VQA v2 val
Vector Quantized Models for Planning
Our key insight is to use discrete autoencoders to capture the multiple possible effects of an action in a stochastic environment.
Machine Translation Decoding beyond Beam Search
Beam search is the go-to method for decoding auto-regressive machine translation models.
Efficient Visual Pretraining with Contrastive Detection
Self-supervised pretraining has been shown to yield powerful representations for transfer learning.
Ranked #48 on
Semantic Segmentation
on Cityscapes val
(using extra training data)
Perceiver: General Perception with Iterative Attention
The perception models used in deep learning on the other hand are designed for individual modalities, often relying on domain-specific assumptions such as the local grid structures exploited by virtually all existing vision models.
Ranked #23 on
Audio Classification
on AudioSet
The MineRL 2020 Competition on Sample Efficient Reinforcement Learning using Human Priors
Although deep reinforcement learning has led to breakthroughs in many difficult domains, these successes have required an ever-increasing number of samples, affording only a shrinking segment of the AI community access to their development.
Solving Mixed Integer Programs Using Neural Networks
Our approach constructs two corresponding neural network-based components, Neural Diving and Neural Branching, to use in a base MIP solver such as SCIP.
AlignNet: Unsupervised Entity Alignment
Recently developed deep learning models are able to learn to segment scenes into component objects without supervision.
Strong Generalization and Efficiency in Neural Programs
We study the problem of learning efficient algorithms that strongly generalize in the framework of neural program induction.
Pointer Graph Networks
This static input structure is often informed purely by insight of the machine learning practitioner, and might not be optimal for the actual task the GNN is solving.
Retrospective Analysis of the 2019 MineRL Competition on Sample Efficient Reinforcement Learning
To facilitate research in the direction of sample efficient reinforcement learning, we held the MineRL Competition on Sample Efficient Reinforcement Learning Using Human Priors at the Thirty-third Conference on Neural Information Processing Systems (NeurIPS 2019).
Low Bit-Rate Speech Coding with VQ-VAE and a WaveNet Decoder
In order to efficiently transmit and store speech signals, speech codecs create a minimally redundant representation of the input signal which is then decoded at the receiver with the best possible perceptual quality.
Unsupervised Doodling and Painting with Improved SPIRAL
We investigate using reinforcement learning agents as generative models of images (extending arXiv:1804. 01118).
Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML
We conclude with a discussion of the rapid learning vs feature reuse question for meta-learning algorithms more broadly.
Generating Diverse High-Fidelity Images with VQ-VAE-2
We explore the use of Vector Quantized Variational AutoEncoder (VQ-VAE) models for large scale image generation.
Classification Accuracy Score for Conditional Generative Models
Deep generative models (DGMs) of images are now sufficiently mature that they produce nearly photorealistic samples and obtain scores similar to the data distribution on heuristics such as Frechet Inception Distance (FID).
Reinforced Genetic Algorithm Learning for Optimizing Computation Graphs
We present a deep reinforcement learning approach to minimizing the execution cost of neural network computation graphs in an optimizing compiler.
Deep reinforcement learning with relational inductive biases
We introduce an approach for augmenting model-free deep reinforcement learning agents with a mechanism for relational reasoning over structured representations, which improves performance, learning efficiency, generalization, and interpretability.
Visual Imitation with a Minimal Adversary
The proposed agent can solve a challenging robot manipulation task of block stacking from only video demonstrations and sparse reward, in which the non-imitating agents fail to learn completely.
Graph Matching Networks for Learning the Similarity of Graph Structured Objects
This paper addresses the challenging problem of retrieval and matching of graph structured objects, and makes two key contributions.
Generating Diverse High-Resolution Images with VQ-VAE
We explore the use of Vector Quantized Variational AutoEncoder (VQ-VAE) models for large scale image generation.
Seeing is Not Necessarily Believing: Limitations of BigGANs for Data Augmentation
In fact, for one model in particular, BigGAN, metrics such as Inception Score or Frechet Inception Distance nearly match those of the dataset, suggesting that these models are close to match-ing the distribution of the training set.
Attentive Neural Processes
Neural Processes (NPs) (Garnelo et al 2018a;b) approach regression by learning to map a context set of observed input-output pairs to a distribution over regression functions.
Preventing Posterior Collapse with delta-VAEs
Due to the phenomenon of "posterior collapse," current latent variable generative models pose a challenging design choice that either weakens the capacity of the decoder or requires augmenting the objective so it does not only maximize the likelihood of the data.
Ranked #7 on
Image Generation
on ImageNet 32x32
(bpd metric)
Generating Diverse Programs with Instruction Conditioned Reinforced Adversarial Learning
In this work, we study the setting in which an agent must learn to generate programs for diverse scenes conditioned on a given symbolic instruction.
Sample Efficient Adaptive Text-to-Speech
Instead, the aim is to produce a network that requires few data at deployment time to rapidly adapt to new speakers.
Deep Audio-Visual Speech Recognition
The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio.
Ranked #6 on
Audio-Visual Speech Recognition
on LRS2
Audio-Visual Speech Recognition
Automatic Speech Recognition (ASR)
+4
Meta-Learning with Latent Embedding Optimization
We show that it is possible to bypass these limitations by learning a data-dependent latent generative representation of model parameters, and performing gradient-based meta-learning in this low-dimensional latent space.
Universal Transformers
Feed-forward and convolutional architectures have recently been shown to achieve superior results on some sequence modeling tasks such as machine translation, with the added advantage that they concurrently process all inputs in the sequence, leading to easy parallelization and faster training times.
Ranked #25 on
Language Modelling
on LAMBADA
Representation Learning with Contrastive Predictive Coding
The key insight of our model is to learn such representations by predicting the future in latent space by using powerful autoregressive models.
Ranked #28 on
Semi-Supervised Image Classification
on ImageNet - 1% labeled data
(Top 5 Accuracy metric)
Representation Learning
Self-Supervised Image Classification
+1
Learning Implicit Generative Models with the Method of Learned Moments
We propose a method of moments (MoM) algorithm for training large-scale implicit generative models.
Relational Deep Reinforcement Learning
We introduce an approach for deep reinforcement learning (RL) that improves upon the efficiency, generalization capacity, and interpretability of conventional approaches through structured perception and relational reasoning.
Relational recurrent neural networks
Memory-based neural networks model temporal data by leveraging an ability to remember information for long periods.
Ranked #54 on
Language Modelling
on WikiText-103
Relational inductive biases, deep learning, and graph networks
As a companion to this paper, we have released an open-source software library for building graph networks, with demonstrations of how to use them in practice.
A Study on Overfitting in Deep Reinforcement Learning
We conclude with a general discussion on overfitting in RL and a study of the generalization behaviors from the perspective of inductive bias.
Synthesizing Programs for Images using Reinforced Adversarial Learning
Advances in deep generative networks have led to impressive results in recent years.
Learning Deep Generative Models of Graphs
Graphs are fundamental data structures which concisely capture the relational structure in many important real-world domains, such as knowledge graphs, physical and social interactions, language, and chemistry.
Memory-based Parameter Adaptation
Deep neural networks have excelled on a wide range of problems, from vision to language and game playing.
Learning to Search with MCTSnets
They are most typically solved by tree search algorithms that simulate ahead into the future, evaluate future states, and back-up those evaluations to the root of a search tree.
Revisiting Bayes by Backprop
We also empirically demonstrate how Bayesian RNNs are superior to traditional RNNs on a language modelling benchmark and an image captioning task, as well as showing how each of these methods improve our model over a variety of other schemes for training them.
Parallel WaveNet: Fast High-Fidelity Speech Synthesis
The recently-developed WaveNet architecture is the current state of the art in realistic speech synthesis, consistently rated as more natural sounding for many different languages than any previous system.
Population Based Training of Neural Networks
Neural networks dominate the modern machine learning landscape, but their training and success still suffer from sensitivity to empirical choices of hyperparameters such as model architecture, loss function, and optimisation algorithm.
Neural Discrete Representation Learning
Learning useful representations without supervision remains a key challenge in machine learning.
Hierarchical Representations for Efficient Architecture Search
We explore efficient neural architecture search methods and show that a simple yet powerful evolutionary algorithm can discover new architectures with excellent performance.
Few-shot Autoregressive Density Estimation: Towards Learning to Learn Distributions
Deep autoregressive models have shown state-of-the-art performance in density estimation for natural images on large-scale datasets such as ImageNet.
StarCraft II: A New Challenge for Reinforcement Learning
Finally, we present initial baseline results for canonical deep reinforcement learning agents applied to the StarCraft II domain.
Ranked #1 on
Starcraft II
on MoveToBeacon
Imagination-Augmented Agents for Deep Reinforcement Learning
We introduce Imagination-Augmented Agents (I2As), a novel architecture for deep reinforcement learning combining model-free and model-based aspects.
Model-based Reinforcement Learning
reinforcement-learning
+1
Learning model-based planning from scratch
Here we introduce the "Imagination-based Planner", the first model-based, sequential decision-making agent that can learn to construct, evaluate, and execute plans.
Metacontrol for Adaptive Imagination-Based Optimization
The metacontroller component is a model-free reinforcement learning agent, which decides both how many iterations of the optimization procedure to run, as well as which model to consult on each iteration.
Bayesian Recurrent Neural Networks
We also empirically demonstrate how Bayesian RNNs are superior to traditional RNNs on a language modelling benchmark and an image captioning task, as well as showing how each of these methods improve our model over a variety of other schemes for training them.
Neural Message Passing for Quantum Chemistry
Supervised learning on molecules has incredible potential to be useful in chemistry, drug discovery, and materials science.
Ranked #4 on
Drug Discovery
on QM9
Neural Episodic Control
Deep reinforcement learning methods attain super-human performance in a wide range of environments.
Understanding Synthetic Gradients and Decoupled Neural Interfaces
When training neural networks, the use of Synthetic Gradients (SG) allows layers or modules to be trained without update locking - without waiting for a true error gradient to be backpropagated - resulting in Decoupled Neural Interfaces (DNIs).
Machine learning prediction errors better than DFT accuracy
We investigate the impact of choosing regressors and molecular representations for the construction of fast machine learning (ML) models of thirteen electronic ground-state properties of organic molecules.
Ranked #17 on
Formation Energy
on QM9
Adversarial Evaluation of Dialogue Models
The recent application of RNN encoder-decoder models has resulted in substantial progress in fully data-driven dialogue systems, but evaluation remains a challenge.
An Online Sequence-to-Sequence Model Using Partial Conditioning
However, they are unsuitable for tasks that require incremental predictions to be made as more data arrives or tasks that have long input sequences and output sequences.
Lip Reading Sentences in the Wild
The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio.
Ranked #4 on
Lipreading
on GRID corpus (mixed-speech)
(using extra training data)
Understanding deep learning requires rethinking generalization
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Connecting Generative Adversarial Networks and Actor-Critic Methods
Both generative adversarial networks (GAN) in unsupervised learning and actor-critic methods in reinforcement learning (RL) have gained a reputation for being difficult to optimize.
Video Pixel Networks
The VPN approaches the best possible performance on the Moving MNIST benchmark, a leap over the previous state of the art, and the generated videos show only minor deviations from the ground truth.
Ranked #1 on
Video Prediction
on KTH
(Cond metric)
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
To improve parallelism and therefore decrease training time, our attention mechanism connects the bottom layer of the decoder to the top layer of the encoder.
Ranked #35 on
Machine Translation
on WMT2014 English-French
Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
Automatically describing the content of an image is a fundamental problem in artificial intelligence that connects computer vision and natural language processing.
WaveNet: A Generative Model for Raw Audio
This paper introduces WaveNet, a deep neural network for generating raw audio waveforms.
Ranked #1 on
Speech Synthesis
on Mandarin Chinese
Decoupled Neural Interfaces using Synthetic Gradients
Training directed neural networks typically requires forward-propagating data through a computation graph, followed by backpropagating error signal, to produce weight updates.
Conditional Image Generation with PixelCNN Decoders
This work explores conditional image generation with a new image density model based on the PixelCNN architecture.
Ranked #11 on
Image Generation
on ImageNet 32x32
(bpd metric)
Strategic Attentive Writer for Learning Macro-Actions
We present a novel deep recurrent neural network architecture that learns to build implicit plans in an end-to-end manner by purely interacting with an environment in reinforcement learning setting.
Matching Networks for One Shot Learning
Our algorithm improves one-shot accuracy on ImageNet from 87. 6% to 93. 2% and from 88. 0% to 93. 8% on Omniglot compared to competing approaches.
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
TensorFlow is an interface for expressing machine learning algorithms, and an implementation for executing such algorithms.
Contextual LSTM (CLSTM) models for Large scale NLP tasks
We evaluate CLSTM on three specific NLP tasks: word prediction, next sentence selection, and sentence topic prediction.
Exploring the Limits of Language Modeling
In this work we explore recent advances in Recurrent Neural Networks for large scale Language Modeling, a task central to language understanding.
Ranked #8 on
Language Modelling
on One Billion Word
Multilingual Language Processing From Bytes
We describe an LSTM-based model which we call Byte-to-Span (BTS) that reads text as bytes and outputs span annotations of the form [start, length, label] where start positions, lengths, and labels are separate entries in our vocabulary.
Order Matters: Sequence to sequence for sets
Sequences have become first class citizens in supervised learning thanks to the resurgence of recurrent neural networks.
Multi-task Sequence to Sequence Learning
This paper examines three multi-task learning (MTL) settings for sequence to sequence models: (a) the oneto-many setting - where the encoder is shared between several tasks such as machine translation and syntactic parsing, (b) the many-to-one setting - useful when only the decoder can be shared, as in the case of translation and image caption generation, and (c) the many-to-many setting - where multiple encoders and decoders are shared, which is the case with unsupervised objectives and translation.
Towards Principled Unsupervised Learning
Supervised learning is successful because it can be solved by the minimization of the training error cost function.
Generating Sentences from a Continuous Space
The standard recurrent neural network language model (RNNLM) generates sentences one word at a time and does not work from an explicit global sentence representation.
A Neural Transducer
However, they are unsuitable for tasks that require incremental predictions to be made as more data arrives or tasks that have long input sequences and output sequences.
Listen, Attend and Spell
Unlike traditional DNN-HMM models, this model learns all the components of a speech recognizer jointly.
A Neural Conversational Model
We find that this straightforward model can generate simple conversations given a large conversational training dataset.
Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
Recurrent Neural Networks can be trained to produce sequences of tokens given some input, as exemplified by recent results in machine translation and image captioning.
Pointer Networks
It differs from the previous attention attempts in that, instead of using attention to blend hidden units of an encoder to a context vector at each decoder step, it uses attention as a pointer to select a member of the input sequence as the output.
Ranked #6 on
Point Cloud Completion
on ShapeNet
(using extra training data)
Beyond Short Snippets: Deep Networks for Video Classification
Convolutional neural networks (CNNs) have been extensively applied for image recognition problems giving state-of-the-art results on recognition, detection, segmentation and retrieval.
Ranked #5 on
Action Recognition
on Sports-1M
Distilling the Knowledge in a Neural Network
A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.
Ranked #5 on
Knowledge Distillation
on ImageNet
Grammar as a Foreign Language
Syntactic constituency parsing is a fundamental problem in natural language processing and has been the subject of intensive research and engineering for decades.
Ranked #23 on
Constituency Parsing
on Penn Treebank
Qualitatively characterizing neural network optimization problems
Training neural networks involves solving large-scale non-convex optimization problems.
Show and Tell: A Neural Image Caption Generator
Experiments on several datasets show the accuracy of the model and the fluency of the language it learns solely from image descriptions.
Ranked #3 on
Image Retrieval with Multi-Modal Query
on MIT-States
Addressing the Rare Word Problem in Neural Machine Translation
Our experiments on the WMT14 English to French translation task show that this method provides a substantial improvement of up to 2. 8 BLEU points over an equivalent NMT system that does not use this technique.
Ranked #40 on
Machine Translation
on WMT2014 English-French
Sequence to Sequence Learning with Neural Networks
Our method uses a multilayered Long Short-Term Memory (LSTM) to map the input sequence to a vector of a fixed dimensionality, and then another deep LSTM to decode the target sequence from the vector.
Ranked #5 on
Traffic Prediction
on PeMS-M
(using extra training data)
Recurrent Neural Network Regularization
We present a simple regularization technique for Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) units.
Ranked #35 on
Language Modelling
on Penn Treebank (Word Level)
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
We evaluate whether features extracted from the activation of a deep convolutional network trained in a fully supervised fashion on a large, fixed set of object recognition tasks can be re-purposed to novel generic tasks.
Why Size Matters: Feature Coding as Nystrom Sampling
Recently, the computer vision and machine learning community has been in favor of feature extraction pipelines that rely on a coding step followed by a linear classifier, due to their overall simplicity, well understood properties of linear classifiers, and their computational efficiency.
Learning with Recursive Perceptual Representations
The use of random projections is key to our method, as we show in the experiments section, in which we observe a consistent improvement over previous --often more complicated-- methods on several vision and speech benchmarks.
Ranked #216 on
Image Classification
on CIFAR-10
