Search Results for author:
Found 17 papers, 0 papers with code
Continuized Acceleration for Quasar Convex Functions in Non-Convex Optimization
Quasar convexity is a condition that allows some first-order methods to efficiently minimize a function even when the optimization landscape is non-convex.
Towards Understanding GD with Hard and Conjugate Pseudo-labels for Test-Time Adaptation
We consider a setting that a model needs to adapt to a new domain under distribution shifts, given that only unlabeled test samples from the new domain are accessible at test time.
Accelerating Hamiltonian Monte Carlo via Chebyshev Integration Time
When the potential $f$ is $L$-smooth and $m$-strongly convex, i. e.\ for sampling from a log-smooth and strongly log-concave target distribution $\pi$, it is known that under a constant integration time, the number of iterations that ideal HMC takes to get an $\epsilon$ Wasserstein-2 distance to the target $\pi$ is $O( \kappa \log \frac{1}{\epsilon} )$, where $\kappa := \frac{L}{m}$ is the condition number.
Provable Acceleration of Heavy Ball beyond Quadratics for a Class of Polyak-Łojasiewicz Functions when the Non-Convexity is Averaged-Out
An additional condition needs to be satisfied for the acceleration result of HB beyond quadratics in this work, which naturally holds when the dimension is one or, more broadly, when the Hessian is diagonal.
No-Regret Dynamics in the Fenchel Game: A Unified Framework for Algorithmic Convex Optimization
We develop an algorithmic framework for solving convex optimization problems using no-regret game dynamics.
Understanding Modern Techniques in Optimization: Frank-Wolfe, Nesterov's Momentum, and Polyak's Momentum
In the first part of this dissertation research, we develop a modular framework that can serve as a recipe for constructing and analyzing iterative algorithms for convex optimization.
Escaping Saddle Points Faster with Stochastic Momentum
At the same time, a widely-observed empirical phenomenon is that in training deep networks stochastic momentum appears to significantly improve convergence time, variants of it have flourished in the development of other popular update methods, e. g. ADAM [KB15], AMSGrad [RKK18], etc.
A Modular Analysis of Provable Acceleration via Polyak's Momentum: Training a Wide ReLU Network and a Deep Linear Network
Our result shows that with the appropriate choice of parameters Polyak's momentum has a rate of $(1-\Theta(\frac{1}{\sqrt{\kappa'}}))^t$.
Understanding How Over-Parametrization Leads to Acceleration: A case of learning a single teacher neuron
Over-parametrization has become a popular technique in deep learning.
Quickly Finding a Benign Region via Heavy Ball Momentum in Non-Convex Optimization
The Heavy Ball Method, proposed by Polyak over five decades ago, is a first-order method for optimizing continuous functions.
Zeroth Order Optimization by a Mixture of Evolution Strategies
Perhaps the only methods that enjoy convergence guarantees are the ones that sample the perturbed points uniformly from a unit sphere or from a multivariate Gaussian distribution with an isotropic covariance.
An Optimistic Acceleration of AMSGrad for Nonconvex Optimization
We propose a new variant of AMSGrad, a popular adaptive gradient based optimization algorithm widely used for training deep neural networks.
Revisiting Projection-Free Optimization for Strongly Convex Constraint Sets
We also show that, for the general case of (smooth) non-convex functions, FW with line search converges with high probability to a stationary point at a rate of $O\left(\frac{1}{t}\right)$, as long as the constraint set is strongly convex -- one of the fastest convergence rates in non-convex optimization.
Acceleration through Optimistic No-Regret Dynamics
In this paper we show that the technique can be enhanced to a rate of $O(1/T^2)$ by extending recent work \cite{RS13, SALS15} that leverages \textit{optimistic learning} to speed up equilibrium computation.
Faster Rates for Convex-Concave Games
We consider the use of no-regret algorithms to compute equilibria for particular classes of convex-concave games.
On Frank-Wolfe and Equilibrium Computation
We consider the Frank-Wolfe (FW) method for constrained convex optimization, and we show that this classical technique can be interpreted from a different perspective: FW emerges as the computation of an equilibrium (saddle point) of a special convex-concave zero sum game.
