Search Results for author:
Found 15 papers, 7 papers with code
SGDR: Stochastic Gradient Descent with Warm Restarts
Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions.
Online Batch Selection for Faster Training of Neural Networks
We investigate online batch selection strategies for two state-of-the-art methods of stochastic gradient-based optimization, AdaDelta and Adam.
Decoupled Weight Decay Regularization
L$_2$ regularization and weight decay regularization are equivalent for standard stochastic gradient descent (when rescaled by the learning rate), but as we demonstrate this is \emph{not} the case for adaptive gradient algorithms, such as Adam.
A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
The original ImageNet dataset is a popular large-scale benchmark for training Deep Neural Networks.
Ranked #1 on
Image Classification
on ImageNet-32
Back to Basics: Benchmarking Canonical Evolution Strategies for Playing Atari
Evolution Strategies (ES) have recently been demonstrated to be a viable alternative to reinforcement learning (RL) algorithms on a set of challenging deep RL problems, including Atari games and MuJoCo humanoid locomotion benchmarks.
Self-Adaptive Surrogate-Assisted Covariance Matrix Adaptation Evolution Strategy
The resulting algorithm, saACM-ES, adjusts online the lifelength of the current surrogate model (the number of CMA-ES generations before learning a new surrogate) and the surrogate hyper-parameters.
Limited-Memory Matrix Adaptation for Large Scale Black-box Optimization
The Covariance Matrix Adaptation Evolution Strategy (CMA-ES) is a popular method to deal with nonconvex and/or stochastic optimization problems when the gradient information is not available.
Anytime Bi-Objective Optimization with a Hybrid Multi-Objective CMA-ES (HMO-CMA-ES)
We propose a multi-objective optimization algorithm aimed at achieving good anytime performance over a wide range of problems.
CMA-ES for Hyperparameter Optimization of Deep Neural Networks
Hyperparameters of deep neural networks are often optimized by grid search, random search or Bayesian optimization.
LM-CMA: an Alternative to L-BFGS for Large Scale Black-box Optimization
Invariance properties of the algorithm do not prevent it from demonstrating a comparable performance to L-BFGS on non-trivial large scale smooth and nonsmooth optimization problems.
Maximum Likelihood-based Online Adaptation of Hyper-parameters in CMA-ES
The Covariance Matrix Adaptation Evolution Strategy (CMA-ES) is widely accepted as a robust derivative-free continuous optimization algorithm for non-linear and non-convex optimization problems.
A Computationally Efficient Limited Memory CMA-ES for Large Scale Optimization
We propose a computationally efficient limited memory Covariance Matrix Adaptation Evolution Strategy for large scale optimization, which we call the LM-CMA-ES.
KL-based Control of the Learning Schedule for Surrogate Black-Box Optimization
This weakness is commonly addressed through surrogate optimization, learning an estimate of the objective function a. k. a.
Fixing Weight Decay Regularization in Adam
We note that common implementations of adaptive gradient algorithms, such as Adam, limit the potential benefit of weight decay regularization, because the weights do not decay multiplicatively (as would be expected for standard weight decay) but by an additive constant factor.
Weight Norm Control
We note that decoupled weight decay regularization is a particular case of weight norm control where the target norm of weights is set to 0.
