Search Results for author:
Found 37 papers, 7 papers with code
Shuffling Momentum Gradient Algorithm for Convex Optimization
The Stochastic Gradient Descent method (SGD) and its stochastic variants have become methods of choice for solving finite-sum optimization problems arising from machine learning and data science thanks to their ability to handle large-scale applications and big datasets.
Sublinear Convergence Rates of Extragradient-Type Methods: A Survey on Classical and Recent Developments
The extragradient (EG), introduced by G. M. Korpelevich in 1976, is a well-known method to approximate solutions of saddle-point problems and their extensions such as variational inequalities and monotone inclusions.
Extragradient-Type Methods with $\mathcal{O} (1/k)$ Last-Iterate Convergence Rates for Co-Hypomonotone Inclusions
We develop two "Nesterov's accelerated" variants of the well-known extragradient method to approximate a solution of a co-hypomonotone inclusion constituted by the sum of two operators, where one is Lipschitz continuous and the other is possibly multivalued.
Randomized Block-Coordinate Optimistic Gradient Algorithms for Root-Finding Problems
In this paper, we develop two new randomized block-coordinate optimistic gradient algorithms to approximate a solution of nonlinear equations in large-scale settings, which are called root-finding problems.
Gradient Descent-Type Methods: Background and Simple Unified Convergence Analysis
In this book chapter, we briefly describe the main components that constitute the gradient descent method and its accelerated and stochastic variants.
Halpern-Type Accelerated and Splitting Algorithms For Monotone Inclusions
In this paper, we develop a new type of accelerated algorithms to solve some classes of maximally monotone equations as well as monotone inclusions.
FedDR -- Randomized Douglas-Rachford Splitting Algorithms for Nonconvex Federated Composite Optimization
These new algorithms can handle statistical and system heterogeneity, which are the two main challenges in federated learning, while achieving the best known communication complexity.
SMG: A Shuffling Gradient-Based Method with Momentum
When the shuffling strategy is fixed, we develop another new algorithm that is similar to existing momentum methods, and prove the same convergence rates for this algorithm under the $L$-smoothness and bounded gradient assumptions.
Convergence Analysis of Homotopy-SGD for non-convex optimization
In this work, we present a first-order stochastic algorithm based on a combination of homotopy methods and SGD, called Homotopy-Stochastic Gradient Descent (H-SGD), which finds interesting connections with some proposed heuristics in the literature, e. g. optimization by Gaussian continuation, training by diffusion, mollifying networks.
Hogwild! over Distributed Local Data Sets with Linearly Increasing Mini-Batch Sizes
We consider big data analysis where training data is distributed among local data sets in a heterogeneous way -- and we wish to move SGD computations to local compute nodes where local data resides.
An Optimal Hybrid Variance-Reduced Algorithm for Stochastic Composite Nonconvex Optimization
In this note we propose a new variant of the hybrid variance-reduced proximal gradient method in [7] to solve a common stochastic composite nonconvex optimization problem under standard assumptions.
Asynchronous Federated Learning with Reduced Number of Rounds and with Differential Privacy from Less Aggregated Gaussian Noise
The feasibility of federated learning is highly constrained by the server-clients infrastructure in terms of network communication.
Hybrid Variance-Reduced SGD Algorithms For Nonconvex-Concave Minimax Problems
This problem class has several computational challenges due to its nonsmoothness, nonconvexity, nonlinearity, and non-separability of the objective functions.
A New Randomized Primal-Dual Algorithm for Convex Optimization with Optimal Last Iterate Rates
We develop a novel unified randomized block-coordinate primal-dual algorithm to solve a class of nonsmooth constrained convex optimization problems, which covers different existing variants and model settings from the literature.
A Hybrid Stochastic Policy Gradient Algorithm for Reinforcement Learning
We propose a novel hybrid stochastic policy gradient estimator by combining an unbiased policy gradient estimator, the REINFORCE estimator, with another biased one, an adapted SARAH estimator for policy optimization.
A Unified Convergence Analysis for Shuffling-Type Gradient Methods
We also study uniformly randomized shuffling variants with different learning rates and model assumptions.
A Newton Frank-Wolfe Method for Constrained Self-Concordant Minimization
We demonstrate how to scalably solve a class of constrained self-concordant minimization problems using linear minimization oracles (LMO) over the constraint set.
Stochastic Gauss-Newton Algorithms for Nonconvex Compositional Optimization
In the expectation case, we establish $\mathcal{O}(\varepsilon^{-2})$ iteration-complexity to achieve a stationary point in expectation and estimate the total number of stochastic oracle calls for both function value and its Jacobian, where $\varepsilon$ is a desired accuracy.
A Hybrid Stochastic Optimization Framework for Stochastic Composite Nonconvex Optimization
We introduce a new approach to develop stochastic optimization algorithms for a class of stochastic composite and possibly nonconvex optimization problems.
Hybrid Stochastic Gradient Descent Algorithms for Stochastic Nonconvex Optimization
We introduce a hybrid stochastic estimator to design stochastic gradient algorithms for solving stochastic optimization problems.
Non-Stationary First-Order Primal-Dual Algorithms with Fast NonErgodic Convergence Rates
By adapting the parameters, we can obtain up to $o(\frac{1}{k})$ convergence rate on the primal objective residuals in nonergodic sense.
Optimization and Control 90C25, 90-08
ProxSARAH: An Efficient Algorithmic Framework for Stochastic Composite Nonconvex Optimization
We also specify the algorithm to the non-composite case that covers existing state-of-the-arts in terms of complexity bounds.
Non-stationary Douglas-Rachford and alternating direction method of multipliers: adaptive stepsizes and convergence
This, in turn, proves the convergence of the method with the new adaptive stepsize rule.
Smooth Primal-Dual Coordinate Descent Algorithms for Nonsmooth Convex Optimization
We propose a new randomized coordinate descent method for a convex optimization template with broad applications.
Sieve-SDP: a simple facial reduction algorithm to preprocess semidefinite programs
We introduce Sieve-SDP, a simple algorithm to preprocess semidefinite programs (SDPs).
Optimization and Control 90-08, 90C22 (Primary) 90C25, 90C06 ( secondary)
Generalized Self-Concordant Functions: A Recipe for Newton-Type Methods
We also achieve both global and local convergence without additional assumption.
Extended Gauss-Newton and ADMM-Gauss-Newton Algorithms for Low-Rank Matrix Optimization
In this paper, we develop a variant of the well-known Gauss-Newton (GN) method to solve a class of nonconvex optimization problems involving low-rank matrix variables.
Convex block-sparse linear regression with expanders -- provably
Our experimental findings on synthetic and real applications support our claims for faster recovery in the convex setting -- as opposed to using dense sensing matrices, while showing a competitive recovery performance.
A single-phase, proximal path-following framework
First, it allows handling non-smooth objectives via proximal operators; this avoids lifting the problem dimension in order to accommodate non-smooth components in optimization.
Adaptive Smoothing Algorithms for Nonsmooth Composite Convex Minimization
We propose an adaptive smoothing algorithm based on Nesterov's smoothing technique in \cite{Nesterov2005c} for solving "fully" nonsmooth composite convex optimization problems.
Structured Sparsity: Discrete and Convex approaches
For each, we present the models in their discrete nature, discuss how to solve the ensuing discrete problems and then describe convex relaxations.
Smooth Alternating Direction Methods for Nonsmooth Constrained Convex Optimization
We propose two new alternating direction methods to solve "fully" nonsmooth constrained convex problems.
Composite convex minimization involving self-concordant-like cost functions
The self-concordant-like property of a smooth convex function is a new analytical structure that generalizes the self-concordant notion.
Constrained convex minimization via model-based excessive gap
We introduce a model-based excessive gap technique to analyze first-order primal- dual methods for constrained convex minimization.
A Primal-Dual Algorithmic Framework for Constrained Convex Minimization
Our main analysis technique provides a fresh perspective on Nesterov's excessive gap technique in a structured fashion and unifies it with smoothing and primal-dual methods.
Scalable sparse covariance estimation via self-concordance
We consider the class of convex minimization problems, composed of a self-concordant function, such as the $\log\det$ metric, a convex data fidelity term $h(\cdot)$ and, a regularizing -- possibly non-smooth -- function $g(\cdot)$.
Composite Self-Concordant Minimization
We propose a variable metric framework for minimizing the sum of a self-concordant function and a possibly non-smooth convex function, endowed with an easily computable proximal operator.
