Search Results for author:
Found 13 papers, 10 papers with code
Training Compute-Optimal Large Language Models
We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget.
Unified Scaling Laws for Routed Language Models
The performance of a language model has been shown to be effectively modeled as a power-law in its parameter count.
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world.
Ranked #1 on
Language Modelling
on StackExchange
Improving language models by retrieving from trillions of tokens
We enhance auto-regressive language models by conditioning on document chunks retrieved from a large corpus, based on local similarity with preceding tokens.
Ranked #1 on
Language Modelling
on WikiText-103
(using extra training data)
Podracer architectures for scalable Reinforcement Learning
Supporting state-of-the-art AI research requires balancing rapid prototyping, ease of use, and quick iteration, with the ability to deploy experiments at a scale traditionally associated with production systems. Deep learning frameworks such as TensorFlow, PyTorch and JAX allow users to transparently make use of accelerators, such as TPUs and GPUs, to offload the more computationally intensive parts of training and inference in modern deep learning systems.
Skillful Precipitation Nowcasting using Deep Generative Models of Radar
To address these challenges, we present a Deep Generative Model for the probabilistic nowcasting of precipitation from radar.
Transformation-based Adversarial Video Prediction on Large-Scale Data
Recent breakthroughs in adversarial generative modeling have led to models capable of producing video samples of high quality, even on large and complex datasets of real-world video.
Ranked #8 on
Video Prediction
on Kinetics-600 12 frames, 64x64
Stabilizing Transformers for Reinforcement Learning
Harnessing the transformer's ability to process long time horizons of information could provide a similar performance boost in partially observable reinforcement learning (RL) domains, but the large-scale transformers used in NLP have yet to be successfully applied to the RL setting.
V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control
Some of the most successful applications of deep reinforcement learning to challenging domains in discrete and continuous control have used policy gradient methods in the on-policy setting.
High Fidelity Speech Synthesis with Adversarial Networks
However, their application in the audio domain has received limited attention, and autoregressive models, such as WaveNet, remain the state of the art in generative modelling of audio signals such as human speech.
Adversarial Video Generation on Complex Datasets
Generative models of natural images have progressed towards high fidelity samples by the strong leveraging of scale.
Ranked #1 on
Video Generation
on Kinetics-600 12 frames, 128x128
3D Character Animation From A Single Photo
Video Generation
+1
TF-Replicator: Distributed Machine Learning for Researchers
We describe TF-Replicator, a framework for distributed machine learning designed for DeepMind researchers and implemented as an abstraction over TensorFlow.
