Search Results for author:
Found 18 papers, 4 papers with code
Acceleration through spectral density estimation
We develop a framework for designing optimal optimization methods in terms of their average-case runtime.
Extra-gradient with player sampling for faster convergence in n-player games
Data-driven modeling increasingly requires to find a Nash equilibrium in multi-player games, e. g. when training GANs.
Universal Asymptotic Optimality of Polyak Momentum
We consider the average-case runtime analysis of algorithms for minimizing quadratic objectives.
SING: A Plug-and-Play DNN Learning Technique
We propose SING (StabIlized and Normalized Gradient), a plug-and-play technique that improves the stability and generalization of the Adam(W) optimizer.
The Curse of Unrolling: Rate of Differentiating Through Optimization
Computing the Jacobian of the solution of an optimization problem is a central problem in machine learning, with applications in hyperparameter optimization, meta-learning, optimization as a layer, and dataset distillation, to name a few.
Only Tails Matter: Average-Case Universality and Robustness in the Convex Regime
The recently developed average-case analysis of optimization methods allows a more fine-grained and representative convergence analysis than usual worst-case results.
Convergence Rates for the MAP of an Exponential Family and Stochastic Mirror Descent -- an Open Problem
We consider the problem of upper bounding the expected log-likelihood sub-optimality of the maximum likelihood estimate (MLE), or a conjugate maximum a posteriori (MAP) for an exponential family, in a non-asymptotic way.
Connecting Sphere Manifolds Hierarchically for Regularization
This paper considers classification problems with hierarchically organized classes.
Acceleration Methods
This monograph covers some recent advances in a range of acceleration techniques frequently used in convex optimization.
Generalization of Quasi-Newton Methods: Application to Robust Symmetric Multisecant Updates
Quasi-Newton techniques approximate the Newton step by estimating the Hessian using the so-called secant equations.
Optimization and Control Numerical Analysis Numerical Analysis
Average-case Acceleration for Bilinear Games and Normal Matrices
First, we show that for zero-sum bilinear games the average-case optimal method is the optimal method for the minimization of the Hamiltonian.
Average-case Acceleration Through Spectral Density Estimation
We develop a framework for the average-case analysis of random quadratic problems and derive algorithms that are optimal under this analysis.
Accelerating Smooth Games by Manipulating Spectral Shapes
Using this perspective, we propose an optimal algorithm for bilinear games.
Extragradient with player sampling for faster Nash equilibrium finding
Data-driven modeling increasingly requires to find a Nash equilibrium in multi-player games, e. g. when training GANs.
Nonlinear Acceleration of CNNs
The Regularized Nonlinear Acceleration (RNA) algorithm is an acceleration method capable of improving the rate of convergence of many optimization schemes such as gradient descend, SAGA or SVRG.
Online Regularized Nonlinear Acceleration
Regularized nonlinear acceleration (RNA) estimates the minimum of a function by post-processing iterates from an algorithm such as the gradient method.
Integration Methods and Optimization Algorithms
We show that accelerated optimization methods can be seen as particular instances of multi-step integration schemes from numerical analysis, applied to the gradient flow equation.
Nonlinear Acceleration of Stochastic Algorithms
Here, we study extrapolation methods in a stochastic setting, where the iterates are produced by either a simple or an accelerated stochastic gradient algorithm.
