Search Results for author:
Found 36 papers, 15 papers with code
Noise-Reuse in Online Evolution Strategies
Online evolution strategies have become an attractive alternative to automatic differentiation (AD) due to their ability to handle chaotic and black-box loss functions, while also allowing more frequent gradient updates than vanilla Evolution Strategies (ES).
General-Purpose In-Context Learning by Meta-Learning Transformers
We further show that the capabilities of meta-trained algorithms are bottlenecked by the accessible state size (memory) determining the next prediction, unlike standard models which are thought to be bottlenecked by parameter count.
Transformer-Based Learned Optimization
We propose a new approach to learned optimization where we represent the computation of an optimizer's update step using a neural network.
VeLO: Training Versatile Learned Optimizers by Scaling Up
While deep learning models have replaced hand-designed features across many domains, these models are still trained with hand-designed optimizers.
Discovered Policy Optimisation
We refer to the immediate result as Learnt Policy Optimisation (LPO).
A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biases
We apply the resulting learned optimizer to a variety of neural network training tasks, where it outperforms the current state of the art learned optimizer -- at matched optimizer computational overhead -- with regard to optimization performance and meta-training speed, and is capable of generalization to tasks far different from those it was meta-trained on.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
Practical tradeoffs between memory, compute, and performance in learned optimizers
We further leverage our analysis to construct a learned optimizer that is both faster and more memory efficient than previous work.
Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategies
Unrolled computation graphs arise in many scenarios, including training RNNs, tuning hyperparameters through unrolled optimization, and training learned optimizers.
Lyapunov Exponents for Diversity in Differentiable Games
We generalize this idea to non-conservative, multi-agent gradient systems by proposing a method - denoted Generalized Ridge Rider (GRR) - for finding arbitrary bifurcation points.
Gradients are Not All You Need
Differentiable programming techniques are widely used in the community and are responsible for the machine learning renaissance of the past several decades.
Learn2Hop: Learned Optimization on Rough Landscapes
Optimization of non-convex loss surfaces containing many local minima remains a critical problem in a variety of domains, including operations research, informatics, and material design.
Training Learned Optimizers with Randomly Initialized Learned Optimizers
We show that a population of randomly initialized learned optimizers can be used to train themselves from scratch in an online fashion, without resorting to a hand designed optimizer in any part of the process.
TaskSet: A Dataset of Optimization Tasks
We present TaskSet, a dataset of tasks for use in training and evaluating optimizers.
Overcoming barriers to the training of effective learned optimizers
In this work we focus on general-purpose learned optimizers capable of training a wide variety of problems with no user-specified hyperparameters.
Parallel Training of Deep Networks with Local Updates
Deep learning models trained on large data sets have been widely successful in both vision and language domains.
Ridge Rider: Finding Diverse Solutions by Following Eigenvectors of the Hessian
In the era of ever decreasing loss functions, SGD and its various offspring have become the go-to optimization tool in machine learning and are a key component of the success of deep neural networks (DNNs).
Reverse engineering learned optimizers reveals known and novel mechanisms
Learned optimizers are algorithms that can themselves be trained to solve optimization problems.
Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves
In this work we focus on general-purpose learned optimizers capable of training a wide variety of problems with no user-specified hyperparameters.
On Linear Identifiability of Learned Representations
Identifiability is a desirable property of a statistical model: it implies that the true model parameters may be estimated to any desired precision, given sufficient computational resources and data.
Using a thousand optimization tasks to learn hyperparameter search strategies
We present TaskSet, a dataset of tasks for use in training and evaluating optimizers.
Towards GAN Benchmarks Which Require Generalization
For many evaluation metrics commonly used as benchmarks for unconditional image generation, trivially memorizing the training set attains a better score than models which are considered state-of-the-art; we consider this problematic.
Learning to Predict Without Looking Ahead: World Models Without Forward Prediction
That useful models can arise out of the messy and slow optimization process of evolution suggests that forward-predictive modeling can arise as a side-effect of optimization under the right circumstances.
Model-based Reinforcement Learning
reinforcement-learning
+1
Learning an Adaptive Learning Rate Schedule
The learning rate is one of the most important hyper-parameters for model training and generalization.
Using learned optimizers to make models robust to input noise
State-of-the art vision models can achieve superhuman performance on image classification tasks when testing and training data come from the same distribution.
Guided Evolutionary Strategies: Escaping the curse of dimensionality in random search
This arises when an approximate gradient is easier to compute than the full gradient (e. g. in meta-learning or unrolled optimization), or when a true gradient is intractable and is replaced with a surrogate (e. g. in certain reinforcement learning applications or training networks with discrete variables).
Learning Unsupervised Learning Rules
Here, our desired task (meta-objective) is the performance of the representation on semi-supervised classification, and we meta-learn an algorithm -- an unsupervised weight update rule -- that produces representations that perform well under this meta-objective.
Learned optimizers that outperform on wall-clock and validation loss
We demonstrate these results on problems where our learned optimizer trains convolutional networks in a fifth of the wall-clock time compared to tuned first-order methods, and with an improvement
Understanding and correcting pathologies in the training of learned optimizers
Deep learning has shown that learned functions can dramatically outperform hand-designed functions on perceptual tasks.
Guided evolutionary strategies: Augmenting random search with surrogate gradients
We propose Guided Evolutionary Strategies, a method for optimally using surrogate gradient directions along with random search.
Meta-Learning Update Rules for Unsupervised Representation Learning
Specifically, we target semi-supervised classification performance, and we meta-learn an algorithm -- an unsupervised weight update rule -- that produces representations useful for this task.
Adversarial Spheres
We hypothesize that this counter intuitive behavior is a naturally occurring result of the high dimensional geometry of the data manifold.
Discrete Sequential Prediction of Continuous Actions for Deep RL
Specifically, we show how Q-values and policies over continuous spaces can be modeled using a next step prediction model over discretized dimensions.
BEGAN: Boundary Equilibrium Generative Adversarial Networks
We propose a new equilibrium enforcing method paired with a loss derived from the Wasserstein distance for training auto-encoder based Generative Adversarial Networks.
Ranked #63 on
Image Generation
on CIFAR-10
(Inception score metric)
Unrolled Generative Adversarial Networks
We introduce a method to stabilize Generative Adversarial Networks (GANs) by defining the generator objective with respect to an unrolled optimization of the discriminator.
Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
In recent years, supervised learning with convolutional networks (CNNs) has seen huge adoption in computer vision applications.
Ranked #9 on
Image Clustering
on Tiny-ImageNet
