Search Results for author:
Found 18 papers, 4 papers with code
From Local SGD to Local Fixed Point Methods for Federated Learning
Most algorithms for solving optimization problems or finding saddle points of convex-concave functions are fixed point algorithms.
Streamlining in the Riemannian Realm: Efficient Riemannian Optimization with Loopless Variance Reduction
These methods replace the outer loop with probabilistic gradient computation triggered by a coin flip in each iteration, ensuring simpler proofs, efficient hyperparameter selection, and sharp convergence guarantees.
MAST: Model-Agnostic Sparsified Training
We introduce a novel optimization problem formulation that departs from the conventional way of minimizing machine learning model loss as a black-box function.
Byzantine Robustness and Partial Participation Can Be Achieved Simultaneously: Just Clip Gradient Differences
Distributed learning has emerged as a leading paradigm for training large machine learning models.
Improving Accelerated Federated Learning with Compression and Importance Sampling
In this setting, the communication between the server and clients poses a major bottleneck.
TAMUNA: Doubly Accelerated Distributed Optimization with Local Training, Compression, and Partial Participation
We propose TAMUNA, the first algorithm for distributed optimization that leveraged the two strategies of local training and compression jointly and allows for partial participation.
Federated Learning with Regularized Client Participation
Under this scheme, each client joins the learning process every $R$ communication rounds, which we refer to as a meta epoch.
Can 5th Generation Local Training Methods Support Client Sampling? Yes!
The celebrated FedAvg algorithm of McMahan et al. (2017) is based on three components: client sampling (CS), data sampling (DS) and local training (LT).
An Optimal Algorithm for Strongly Convex Min-min Optimization
In this paper we study the smooth strongly convex minimization problem $\min_{x}\min_y f(x, y)$.
Minibatch Stochastic Three Points Method for Unconstrained Smooth Minimization
In this paper, we propose a new zero order optimization method called minibatch stochastic three points (MiSTP) method to solve an unconstrained minimization problem in a setting where only an approximation of the objective function evaluation is possible.
Variance Reduced ProxSkip: Algorithm, Theory and Application to Federated Learning
We study distributed optimization methods based on the {\em local training (LT)} paradigm: achieving communication efficiency by performing richer local gradient-based training on the clients before parameter averaging.
Federated Optimization Algorithms with Random Reshuffling and Gradient Compression
To reveal the true advantages of RR in the distributed learning with compression, we propose a new method called DIANA-RR that reduces the compression variance and has provably better convergence rates than existing counterparts with with-replacement sampling of stochastic gradients.
Federated Random Reshuffling with Compression and Variance Reduction
Random Reshuffling (RR), which is a variant of Stochastic Gradient Descent (SGD) employing sampling without replacement, is an immensely popular method for training supervised machine learning models via empirical risk minimization.
ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!
The canonical approach to solving such problems is via the proximal gradient descent (ProxGD) algorithm, which is based on the evaluation of the gradient of $f$ and the prox operator of $\psi$ in each iteration.
Server-Side Stepsizes and Sampling Without Replacement Provably Help in Federated Optimization
Together, our results on the advantage of large and small server-side stepsizes give a formal justification for the practice of adaptive server-side optimization in federated learning.
Random Reshuffling with Variance Reduction: New Analysis and Better Rates
One of the tricks that works so well in practice that it is used as default in virtually all widely used machine learning software is {\em random reshuffling (RR)}.
Distributed Proximal Splitting Algorithms with Rates and Acceleration
We analyze several generic proximal splitting algorithms well suited for large-scale convex nonsmooth optimization.
From Local SGD to Local Fixed-Point Methods for Federated Learning
Most algorithms for solving optimization problems or finding saddle points of convex-concave functions are fixed-point algorithms.
