Search Results for author:
Found 27 papers, 2 papers with code
Optimization on a Finer Scale: Bounded Local Subgradient Variation Perspective
The resulting class of objective functions encapsulates the classes of objective functions traditionally studied in optimization, which are defined based on either Lipschitz continuity of the objective or H\"{o}lder/Lipschitz continuity of its gradient.
A Primal-Dual Algorithm for Faster Distributionally Robust Optimization
We consider the penalized distributionally robust optimization (DRO) problem with a closed, convex uncertainty set, a setting that encompasses the $f$-DRO, Wasserstein-DRO, and spectral/$L$-risk formulations used in practice.
Robust Second-Order Nonconvex Optimization and Its Application to Low Rank Matrix Sensing
We introduce a general framework for efficiently finding an approximate SOSP with \emph{dimension-independent} accuracy guarantees, using $\widetilde{O}({D^2}/{\epsilon})$ samples where $D$ is the ambient dimension and $\epsilon$ is the fraction of corrupted datapoints.
Last Iterate Convergence of Incremental Methods and Applications in Continual Learning
We further obtain generalizations of our results to weighted averaging of the iterates with increasing weights, which can be seen as interpolating between the last iterate and the average iterate guarantees.
Robustly Learning Single-Index Models via Alignment Sharpness
We give an efficient learning algorithm, achieving a constant factor approximation to the optimal loss, that succeeds under a range of distributions (including log-concave distributions) and a broad class of monotone and Lipschitz link functions.
Variance Reduced Halpern Iteration for Finite-Sum Monotone Inclusions
Our main contributions are variants of the classical Halpern iteration that employ variance reduction to obtain improved complexity guarantees in which $n$ component operators in the finite sum are ``on average'' either cocoercive or Lipschitz continuous and monotone, with parameter $L$.
Information-Computation Tradeoffs for Learning Margin Halfspaces with Random Classification Noise
Our main result is a lower bound for Statistical Query (SQ) algorithms and low-degree polynomial tests suggesting that the quadratic dependence on $1/\epsilon$ in the sample complexity is inherent for computationally efficient algorithms.
Empirical Risk Minimization with Shuffled SGD: A Primal-Dual Perspective and Improved Bounds
Contrary to the empirical practice of sampling from the datasets without replacement and with (possible) reshuffling at each epoch, the theoretical counterpart of SGD usually relies on the assumption of sampling with replacement.
Robustly Learning a Single Neuron via Sharpness
We study the problem of learning a single neuron with respect to the $L_2^2$-loss in the presence of adversarial label noise.
Accelerated Cyclic Coordinate Dual Averaging with Extrapolation for Composite Convex Optimization
Exploiting partial first-order information in a cyclic way is arguably the most natural strategy to obtain scalable first-order methods.
Cyclic Block Coordinate Descent With Variance Reduction for Composite Nonconvex Optimization
Our convergence analysis is based on a gradient Lipschitz condition with respect to a Mahalanobis norm, inspired by a recent progress on cyclic block coordinate methods.
Stochastic Halpern Iteration with Variance Reduction for Stochastic Monotone Inclusions
We study stochastic monotone inclusion problems, which widely appear in machine learning applications, including robust regression and adversarial learning.
A Fast Scale-Invariant Algorithm for Non-negative Least Squares with Non-negative Data
In particular, while the oracle complexity of unconstrained least squares problems necessarily scales with one of the data matrix constants (typically the spectral norm) and these problems are solved to additive error, we show that nonnegative least squares problems with nonnegative data are solvable to multiplicative error and with complexity that is independent of any matrix constants.
Coordinate Linear Variance Reduction for Generalized Linear Programming
\textsc{clvr} yields improved complexity results for (GLP) that depend on the max row norm of the linear constraint matrix in (GLP) rather than the spectral norm.
Variance Reduction via Primal-Dual Accelerated Dual Averaging for Nonsmooth Convex Finite-Sums
We study structured nonsmooth convex finite-sum optimization that appears widely in machine learning applications, including support vector machines and least absolute deviation.
Cyclic Coordinate Dual Averaging with Extrapolation
This class includes composite convex optimization problems and convex-concave min-max optimization problems as special cases and has not been addressed by the existing work.
Parameter-free Locally Accelerated Conditional Gradients
Projection-free conditional gradient (CG) methods are the algorithms of choice for constrained optimization setups in which projections are often computationally prohibitive but linear optimization over the constraint set remains computationally feasible.
Potential Function-based Framework for Making the Gradients Small in Convex and Min-Max Optimization
We introduce a novel potential function-based framework to study the convergence of standard methods for making the gradients small in smooth convex optimization and convex-concave min-max optimization.
Complementary Composite Minimization, Small Gradients in General Norms, and Applications
We introduce a new algorithmic framework for complementary composite minimization, where the objective function decouples into a (weakly) smooth and a uniformly convex term.
Efficient Methods for Structured Nonconvex-Nonconcave Min-Max Optimization
The use of min-max optimization in adversarial training of deep neural network classifiers and training of generative adversarial networks has motivated the study of nonconvex-nonconcave optimization objectives, which frequently arise in these applications.
Halpern Iteration for Near-Optimal and Parameter-Free Monotone Inclusion and Strong Solutions to Variational Inequalities
We leverage the connections between nonexpansive maps, monotone Lipschitz operators, and proximal mappings to obtain near-optimal (i. e., optimal up to poly-log factors in terms of iteration complexity) and parameter-free methods for solving monotone inclusion problems.
Conjugate Gradients and Accelerated Methods Unified: The Approximate Duality Gap View
This note provides a novel, simple analysis of the method of conjugate gradients for the minimization of convex quadratic functions.
Locally Accelerated Conditional Gradients
As such, they are frequently used in solving smooth convex optimization problems over polytopes, for which the computational cost of orthogonal projections would be prohibitive.
Generalized Momentum-Based Methods: A Hamiltonian Perspective
We take a Hamiltonian-based perspective to generalize Nesterov's accelerated gradient descent and Polyak's heavy ball method to a broad class of momentum methods in the setting of (possibly) constrained minimization in Euclidean and non-Euclidean normed vector spaces.
Langevin Monte Carlo without smoothness
Langevin Monte Carlo (LMC) is an iterative algorithm used to generate samples from a distribution that is known only up to a normalizing constant.
Lower Bounds for Parallel and Randomized Convex Optimization
We study the question of whether parallelization in the exploration of the feasible set can be used to speed up convex optimization, in the local oracle model of computation.
Alternating Randomized Block Coordinate Descent
While various block-coordinate-descent-type methods have been studied extensively, only alternating minimization – which applies to the setting of only two blocks – is known to have convergence time that scales independently of the least smooth block.
