Search Results for author:
Found 28 papers, 5 papers with code
From Local SGD to Local Fixed Point Methods for Federated Learning
Most algorithms for solving optimization problems or finding saddle points of convex-concave functions are fixed point algorithms.
Acceleration for Compressed Gradient Descent in Distributed Optimization
Due to the high communication cost in distributed and federated learning problems, methods relying on sparsification or quantization of communicated messages are becoming increasingly popular.
An Optimal Algorithm for Strongly Convex Min-min Optimization
In this paper we study the smooth strongly convex minimization problem $\min_{x}\min_y f(x, y)$.
Smooth Monotone Stochastic Variational Inequalities and Saddle Point Problems: A Survey
This paper is a survey of methods for solving smooth (strongly) monotone stochastic variational inequalities.
Communication Acceleration of Local Gradient Methods via an Accelerated Primal-Dual Algorithm with Inexact Prox
Inspired by a recent breakthrough of Mishchenko et al (2022), who for the first time showed that local gradient steps can lead to provable communication acceleration, we propose an alternative algorithm which obtains the same communication acceleration as their method (ProxSkip).
On Scaled Methods for Saddle Point Problems
Methods with adaptive scaling of different features play a key role in solving saddle point problems, primarily due to Adam's popularity for solving adversarial machine learning problems, including GANS training.
Optimal Gradient Sliding and its Application to Distributed Optimization Under Similarity
Finally the method is extended to distributed saddle-problems (under function similarity) by means of solving a class of variational inequalities, achieving lower communication and computation complexity bounds.
The First Optimal Acceleration of High-Order Methods in Smooth Convex Optimization
Arjevani et al. (2019) established the lower bound $\Omega\left(\epsilon^{-2/(3p+1)}\right)$ on the number of the $p$-th order oracle calls required by an algorithm to find an $\epsilon$-accurate solution to the problem, where the $p$-th order oracle stands for the computation of the objective function value and the derivatives up to the order $p$.
The First Optimal Algorithm for Smooth and Strongly-Convex-Strongly-Concave Minimax Optimization
However, the existing state-of-the-art methods do not match this lower bound: algorithms of Lin et al. (2020) and Wang and Li (2020) have gradient evaluation complexity $\mathcal{O}\left( \sqrt{\kappa_x\kappa_y}\log^3\frac{1}{\epsilon}\right)$ and $\mathcal{O}\left( \sqrt{\kappa_x\kappa_y}\log^3 (\kappa_x\kappa_y)\log\frac{1}{\epsilon}\right)$, respectively.
Similarity learning for wells based on logging data
The essence of the interwell correlation constitutes an assessment of the similarities between geological profiles.
Optimal Algorithms for Decentralized Stochastic Variational Inequalities
Our algorithms are the best among the available literature not only in the decentralized stochastic case, but also in the decentralized deterministic and non-distributed stochastic cases.
Accelerated Primal-Dual Gradient Method for Smooth and Convex-Concave Saddle-Point Problems with Bilinear Coupling
In this paper we study the convex-concave saddle-point problem $\min_x \max_y f(x) + y^T \mathbf{A} x - g(y)$, where $f(x)$ and $g(y)$ are smooth and convex functions.
Lower Bounds and Optimal Algorithms for Smooth and Strongly Convex Decentralized Optimization Over Time-Varying Networks
We consider the task of minimizing the sum of smooth and strongly convex functions stored in a decentralized manner across the nodes of a communication network whose links are allowed to change in time.
An Optimal Algorithm for Strongly Convex Minimization under Affine Constraints
Optimization problems under affine constraints appear in various areas of machine learning.
Optimization and Control
ADOM: Accelerated Decentralized Optimization Method for Time-Varying Networks
We propose ADOM - an accelerated method for smooth and strongly convex decentralized optimization over time-varying networks.
IntSGD: Adaptive Floatless Compression of Stochastic Gradients
We propose a family of adaptive integer compression operators for distributed Stochastic Gradient Descent (SGD) that do not communicate a single float.
Decentralized Distributed Optimization for Saddle Point Problems
We consider distributed convex-concave saddle point problems over arbitrary connected undirected networks and propose a decentralized distributed algorithm for their solution.
Distributed Optimization
Optimization and Control
Distributed, Parallel, and Cluster Computing
A Linearly Convergent Algorithm for Decentralized Optimization: Sending Less Bits for Free!
Decentralized optimization methods enable on-device training of machine learning models without a central coordinator.
Linearly Converging Error Compensated SGD
Moreover, using our general scheme, we develop new variants of SGD that combine variance reduction or arbitrary sampling with error feedback and quantization and derive the convergence rates for these methods beating the state-of-the-art results.
Optimal and Practical Algorithms for Smooth and Strongly Convex Decentralized Optimization
We propose two new algorithms for this decentralized optimization problem and equip them with complexity guarantees.
From Local SGD to Local Fixed-Point Methods for Federated Learning
Most algorithms for solving optimization problems or finding saddle points of convex-concave functions are fixed-point algorithms.
Acceleration for Compressed Gradient Descent in Distributed and Federated Optimization
Due to the high communication cost in distributed and federated learning problems, methods relying on compression of communicated messages are becoming increasingly popular.
Variance Reduced Coordinate Descent with Acceleration: New Method With a Surprising Application to Finite-Sum Problems
We propose an accelerated version of stochastic variance reduced coordinate descent -- ASVRCD.
Distributed Fixed Point Methods with Compressed Iterates
We propose basic and natural assumptions under which iterative optimization methods with compressed iterates can be analyzed.
Stochastic Newton and Cubic Newton Methods with Simple Local Linear-Quadratic Rates
We present two new remarkably simple stochastic second-order methods for minimizing the average of a very large number of sufficiently smooth and strongly convex functions.
Stochastic Proximal Langevin Algorithm: Potential Splitting and Nonasymptotic Rates
We propose a new algorithm---Stochastic Proximal Langevin Algorithm (SPLA)---for sampling from a log concave distribution.
Revisiting Stochastic Extragradient
We fix a fundamental issue in the stochastic extragradient method by providing a new sampling strategy that is motivated by approximating implicit updates.
Don't Jump Through Hoops and Remove Those Loops: SVRG and Katyusha are Better Without the Outer Loop
A key structural element in both of these methods is the inclusion of an outer loop at the beginning of which a full pass over the training data is made in order to compute the exact gradient, which is then used to construct a variance-reduced estimator of the gradient.
