Search Results for author:
Found 33 papers, 14 papers with code
Sharpness-Aware Minimization: General Analysis and Improved Rates
Sharpness-Aware Minimization (SAM) has emerged as a powerful method for improving generalization in machine learning models by minimizing the sharpness of the loss landscape.
Multiplayer Federated Learning: Reaching Equilibrium with Less Communication
Traditional Federated Learning (FL) approaches assume collaborative clients with aligned objectives working towards a shared global model.
Stochastic Polyak Step-sizes and Momentum: Convergence Guarantees and Practical Performance
Our convergence analysis of SHB is tight and obtains the convergence guarantees of stochastic Polyak step-size for SGD as a special case.
Dissipative Gradient Descent Ascent Method: A Control Theory Inspired Algorithm for Min-max Optimization
Gradient Descent Ascent (GDA) methods for min-max optimization problems typically produce oscillatory behavior that can lead to instability, e. g., in bilinear settings.
Stochastic Extragradient with Random Reshuffling: Improved Convergence for Variational Inequalities
The Stochastic Extragradient (SEG) method is one of the most popular algorithms for solving finite-sum min-max optimization and variational inequality problems (VIPs) appearing in various machine learning tasks.
Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGrad
Adaptive methods are extremely popular in machine learning as they make learning rate tuning less expensive.
Locally Adaptive Federated Learning
In this work, we propose locally adaptive federated learning algorithms, that leverage the local geometric information for each client function.
Communication-Efficient Gradient Descent-Accent Methods for Distributed Variational Inequalities: Unified Analysis and Local Updates
However, with the increase of minimax optimization and variational inequality problems in machine learning, the necessity of designing efficient distributed/federated learning approaches for these problems is becoming more apparent.
Single-Call Stochastic Extragradient Methods for Structured Non-monotone Variational Inequalities: Improved Analysis under Weaker Conditions
In addition, several important questions regarding the convergence properties of these methods are still open, including mini-batching, efficient step-size selection, and convergence guarantees under different sampling strategies.
A Unified Approach to Reinforcement Learning, Quantal Response Equilibria, and Two-Player Zero-Sum Games
This work studies an algorithm, which we call magnetic mirror descent, that is inspired by mirror descent and the non-Euclidean proximal gradient algorithm.
Stochastic Gradient Descent-Ascent: Unified Theory and New Efficient Methods
Although variants of the new methods are known for solving minimization problems, they were never considered or analyzed for solving min-max problems and VIPs.
Stochastic Extragradient: General Analysis and Improved Rates
The Stochastic Extragradient (SEG) method is one of the most popular algorithms for solving min-max optimization and variational inequalities problems (VIP) appearing in various machine learning tasks.
Stochastic Mirror Descent: Convergence Analysis and Adaptive Variants via the Mirror Stochastic Polyak Stepsize
We investigate the convergence of stochastic mirror descent (SMD) under interpolation in relatively smooth and smooth convex optimization.
Extragradient Method: $O(1/K)$ Last-Iterate Convergence for Monotone Variational Inequalities and Connections With Cocoercivity
In this paper, we resolve one of such questions and derive the first last-iterate $O(1/K)$ convergence rate for EG for monotone and Lipschitz VIP without any additional assumptions on the operator unlike the only known result of this type (Golowich et al., 2020) that relies on the Lipschitzness of the Jacobian of the operator.
On the Convergence of Stochastic Extragradient for Bilinear Games using Restarted Iteration Averaging
We study the stochastic bilinear minimax optimization problem, presenting an analysis of the same-sample Stochastic ExtraGradient (SEG) method with constant step size, and presenting variations of the method that yield favorable convergence.
Stochastic Gradient Descent-Ascent and Consensus Optimization for Smooth Games: Convergence Analysis under Expected Co-coercivity
Two of the most prominent algorithms for solving unconstrained smooth games are the classical stochastic gradient descent-ascent (SGDA) and the recently introduced stochastic consensus optimization (SCO) [Mescheder et al., 2017].
AI-SARAH: Adaptive and Implicit Stochastic Recursive Gradient Methods
We present AI-SARAH, a practical variant of SARAH.
Stochastic Hamiltonian Gradient Methods for Smooth Games
The success of adversarial formulations in machine learning has brought renewed motivation for smooth games.
Unified Analysis of Stochastic Gradient Methods for Composite Convex and Smooth Optimization
We showcase this by obtaining a simple formula for the optimal minibatch size of two variance reduced methods (\textit{L-SVRG} and \textit{SAGA}).
SGD for Structured Nonconvex Functions: Learning Rates, Minibatching and Interpolation
Stochastic Gradient Descent (SGD) is being used routinely for optimizing non-convex functions.
A Unified Theory of Decentralized SGD with Changing Topology and Local Updates
Decentralized stochastic optimization methods have gained a lot of attention recently, mainly because of their cheap per iteration cost, data locality, and their communication-efficiency.
Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence
Consequently, the proposed stochastic Polyak step-size (SPS) is an attractive choice for setting the learning rate for stochastic gradient descent (SGD).
Randomized Iterative Methods for Linear Systems: Momentum, Inexactness and Gossip
In the era of big data, one of the key challenges is the development of novel optimization algorithms that can accommodate vast amounts of data while at the same time satisfying constraints and limitations of the problem under study.
Revisiting Randomized Gossip Algorithms: General Framework, Convergence Rates and Novel Block and Accelerated Protocols
In this work we present a new framework for the analysis and design of randomized gossip algorithms for solving the average consensus problem.
Convergence Analysis of Inexact Randomized Iterative Methods
We relax this requirement by allowing for the sub-problem to be solved inexactly.
SGD: General Analysis and Improved Rates
By specializing our theorem to different mini-batching strategies, such as sampling with replacement and independent sampling, we derive exact expressions for the stepsize as a function of the mini-batch size.
A Privacy Preserving Randomized Gossip Algorithm via Controlled Noise Insertion
In this work we present a randomized gossip algorithm for solving the average consensus problem while at the same time protecting the information about the initial private values stored at the nodes.
Stochastic Gradient Push for Distributed Deep Learning
Distributed data-parallel algorithms aim to accelerate the training of deep neural networks by parallelizing the computation of large mini-batch gradient updates across multiple nodes.
Provably Accelerated Randomized Gossip Algorithms
In this work we present novel provably accelerated gossip algorithms for solving the average consensus problem.
Accelerated Gossip via Stochastic Heavy Ball Method
In this paper we show how the stochastic heavy ball method (SHB) -- a popular method for solving stochastic convex and non-convex optimization problems --operates as a randomized gossip algorithm.
Momentum and Stochastic Momentum for Stochastic Gradient, Newton, Proximal Point and Subspace Descent Methods
We prove linear convergence of several stochastic methods with stochastic momentum, and show that in some sparse data regimes and for sufficiently small momentum parameters, these methods enjoy better overall complexity than methods with deterministic momentum.
Linearly convergent stochastic heavy ball method for minimizing generalization error
In this work we establish the first linear convergence result for the stochastic heavy ball method.
Privacy Preserving Randomized Gossip Algorithms
In this work we present three different randomized gossip algorithms for solving the average consensus problem while at the same time protecting the information about the initial private values stored at the nodes.
Optimization and Control
