Search Results for author:
Found 186 papers, 34 papers with code
Collaborative Value Function Estimation Under Model Mismatch: A Federated Temporal Difference Analysis
Federated reinforcement learning (FedRL) enables collaborative learning while preserving data privacy by preventing direct data exchange between agents.
BurTorch: Revisiting Training from First Principles by Coupling Autodiff, Math Optimization, and Systems
For small compute graphs, BurTorch outperforms best-practice solutions by up to $\times 2000$ in runtime and reduces memory consumption by up to $\times 3500$.
Smoothed Normalization for Efficient Distributed Private Optimization
The reason is that standard privacy techniques require bounding the participants' contributions, usually enforced via $\textit{clipping}$ of the updates.
A Novel Unified Parametric Assumption for Nonconvex Optimization
Nonconvex optimization is central to modern machine learning, but the general framework of nonconvex optimization yields weak convergence guarantees that are too pessimistic compared to practice.
The Ball-Proximal (="Broximal") Point Method: a New Algorithm, Convergence Theory, and Applications
Non-smooth and non-convex global optimization poses significant challenges across various applications, where standard gradient-based methods often struggle.
ATA: Adaptive Task Allocation for Efficient Resource Management in Distributed Machine Learning
Asynchronous methods are fundamental for parallelizing computations in distributed machine learning.
Symmetric Pruning of Large Language Models
Popular post-training pruning methods such as Wanda and RIA are known for their simple, yet effective, designs that have shown exceptional empirical performance.
Ringmaster ASGD: The First Asynchronous SGD with Optimal Time Complexity
We establish, through rigorous theoretical analysis, that Ringmaster ASGD achieves optimal time complexity under arbitrarily heterogeneous and dynamically fluctuating worker computation times.
On the Convergence of DP-SGD with Adaptive Clipping
This paper provides the first comprehensive convergence analysis of SGD with quantile clipping (QC-SGD).
MARINA-P: Superior Performance in Non-smooth Federated Optimization with Adaptive Stepsizes
Recent advancements have primarily focused on smooth convex and non-convex regimes, leaving a significant gap in understanding the non-smooth convex setting.
Differentially Private Random Block Coordinate Descent
Coordinate Descent (CD) methods have gained significant attention in machine learning due to their effectiveness in solving high-dimensional problems and their ability to decompose complex optimization tasks.
Methods with Local Steps and Random Reshuffling for Generally Smooth Non-Convex Federated Optimization
Many existing algorithms designed for standard smooth problems need to be revised.
Pushing the Limits of Large Language Model Quantization via the Linearity Theorem
Quantizing large language models has become a standard way to reduce their memory and computational costs.
Error Feedback under $(L_0,L_1)$-Smoothness: Normalization and Momentum
Moreover, to the best of our knowledge, all existing analyses under generalized smoothness either i) focus on single-node settings or ii) make unrealistically strong assumptions for distributed settings, such as requiring data heterogeneity, and almost surely bounded stochastic gradient noise variance.
Tighter Performance Theory of FedExProx
We revisit FedExProx - a recently proposed distributed optimization method designed to enhance convergence properties of parallel proximal algorithms via extrapolation.
Unlocking FedNL: Self-Contained Compute-Optimized Implementation
Federated Learning (FL) is an emerging paradigm that enables intelligent agents to collaboratively train Machine Learning (ML) models in a distributed manner, eliminating the need for sharing their local data.
Randomized Asymmetric Chain of LoRA: The First Meaningful Theoretical Framework for Low-Rank Adaptation
One of the most widely used methods is Low-Rank Adaptation (LoRA), with adaptation update expressed as the product of two low-rank matrices.
MindFlayer: Efficient Asynchronous Parallel SGD in the Presence of Heterogeneous and Random Worker Compute Times
We study the problem of minimizing the expectation of smooth nonconvex functions with the help of several parallel workers whose role is to compute stochastic gradients.
On the Convergence of FedProx with Extrapolation and Inexact Prox
Enhancing the FedProx federated learning algorithm (Li et al., 2020) with server-side extrapolation, Li et al. (2024a) recently introduced the FedExProx method.
Methods for Convex $(L_0,L_1)$-Smooth Optimization: Clipping, Acceleration, and Adaptivity
We also extend these results to the stochastic case under the over-parameterization assumption, propose a new accelerated method for convex $(L_0, L_1)$-smooth optimization, and derive new convergence rates for Adaptive Gradient Descent (Malitsky and Mishchenko, 2020).
Cohort Squeeze: Beyond a Single Communication Round per Cohort in Cross-Device Federated Learning
Virtually all federated learning (FL) methods, including FedAvg, operate in the following manner: i) an orchestrating server sends the current model parameters to a cohort of clients selected via certain rule, ii) these clients then independently perform a local training procedure (e. g., via SGD or Adam) using their own training data, and iii) the resulting models are shipped to the server for aggregation.
Sparse-ProxSkip: Accelerated Sparse-to-Sparse Training in Federated Learning
In Federated Learning (FL), both client resource constraints and communication costs pose major problems for training large models.
SPAM: Stochastic Proximal Point Method with Momentum Variance Reduction for Non-convex Cross-Device Federated Learning
Cross-device training is a crucial subfield of federated learning, where the number of clients can reach into the billions.
Freya PAGE: First Optimal Time Complexity for Large-Scale Nonconvex Finite-Sum Optimization with Heterogeneous Asynchronous Computations
In practical distributed systems, workers are typically not homogeneous, and due to differences in hardware configurations and network conditions, can have highly varying processing times.
A Unified Theory of Stochastic Proximal Point Methods without Smoothness
This paper presents a comprehensive analysis of a broad range of variations of the stochastic proximal point method (SPPM).
FedP3: Federated Personalized and Privacy-friendly Network Pruning under Model Heterogeneity
The interest in federated learning has surged in recent research due to its unique ability to train a global model using privacy-secured information held locally on each client.
FedComLoc: Communication-Efficient Distributed Training of Sparse and Quantized Models
Federated Learning (FL) has garnered increasing attention due to its unique characteristic of allowing heterogeneous clients to process their private data locally and interact with a central server, while being respectful of privacy.
Streamlining in the Riemannian Realm: Efficient Riemannian Optimization with Loopless Variance Reduction
These methods replace the outer loop with probabilistic gradient computation triggered by a coin flip in each iteration, ensuring simpler proofs, efficient hyperparameter selection, and sharp convergence guarantees.
LoCoDL: Communication-Efficient Distributed Learning with Local Training and Compression
In Distributed optimization and Learning, and even more in the modern framework of federated learning, communication, which is slow and costly, is critical.
Error Feedback Reloaded: From Quadratic to Arithmetic Mean of Smoothness Constants
Error Feedback (EF) is a highly popular and immensely effective mechanism for fixing convergence issues which arise in distributed training methods (such as distributed GD or SGD) when these are enhanced with greedy communication compression techniques such as TopK.
Improving the Worst-Case Bidirectional Communication Complexity for Nonconvex Distributed Optimization under Function Similarity
We introduce M3, a method combining MARINA-P with uplink compression and a momentum step, achieving bidirectional compression with provable improvements in total communication complexity as the number of workers increases.
Shadowheart SGD: Distributed Asynchronous SGD with Optimal Time Complexity Under Arbitrary Computation and Communication Heterogeneity
We consider nonconvex stochastic optimization problems in the asynchronous centralized distributed setup where the communication times from workers to a server can not be ignored, and the computation and communication times are potentially different for all workers.
Correlated Quantization for Faster Nonconvex Distributed Optimization
We analyze the forefront distributed non-convex optimization algorithm MARINA (Gorbunov et al., 2022) utilizing the proposed correlated quantizers and show that it outperforms the original MARINA and distributed SGD of Suresh et al. (2022) with regard to the communication complexity.
Kimad: Adaptive Gradient Compression with Bandwidth Awareness
In distributed training, communication often emerges as a bottleneck.
MAST: Model-Agnostic Sparsified Training
We introduce a novel optimization problem formulation that departs from the conventional way of minimizing machine learning model loss as a black-box function.
Byzantine Robustness and Partial Participation Can Be Achieved at Once: Just Clip Gradient Differences
Distributed learning has emerged as a leading paradigm for training large machine learning models.
Communication Compression for Byzantine Robust Learning: New Efficient Algorithms and Improved Rates
Byzantine robustness is an essential feature of algorithms for certain distributed optimization problems, typically encountered in collaborative/federated learning.
High-Probability Convergence for Composite and Distributed Stochastic Minimization and Variational Inequalities with Heavy-Tailed Noise
High-probability analysis of stochastic first-order optimization methods under mild assumptions on the noise has been gaining a lot of attention in recent years.
Towards a Better Theoretical Understanding of Independent Subnetwork Training
We identify fundamental differences between IST and alternative approaches, such as distributed methods with compressed communication, and provide a precise analysis of its optimization performance on a quadratic model.
Understanding Progressive Training Through the Framework of Randomized Coordinate Descent
We propose a Randomized Progressive Training algorithm (RPT) -- a stochastic proxy for the well-known Progressive Training method (PT) (Karras et al., 2017).
Improving Accelerated Federated Learning with Compression and Importance Sampling
In this setting, the communication between the server and clients poses a major bottleneck.
Clip21: Error Feedback for Gradient Clipping
Motivated by the increasing popularity and importance of large-scale training under differential privacy (DP) constraints, we study distributed gradient methods with gradient clipping, i. e., clipping applied to the gradients computed from local information at the nodes.
Global-QSGD: Practical Floatless Quantization for Distributed Learning with Theoretical Guarantees
To obtain theoretical guarantees, we generalize the notion of standard unbiased compression operators to incorporate Global-QSGD.
Error Feedback Shines when Features are Rare
To illustrate our main result, we show that in order to find a random vector $\hat{x}$ such that $\lVert {\nabla f(\hat{x})} \rVert^2 \leq \varepsilon$ in expectation, ${\color{green}\sf GD}$ with the ${\color{green}\sf Top1}$ sparsifier and ${\color{green}\sf EF}$ requires ${\cal O} \left(\left( L+{\color{blue}r} \sqrt{ \frac{{\color{red}c}}{n} \min \left( \frac{{\color{red}c}}{n} \max_i L_i^2, \frac{1}{n}\sum_{i=1}^n L_i^2 \right) }\right) \frac{1}{\varepsilon} \right)$ bits to be communicated by each worker to the server only, where $L$ is the smoothness constant of $f$, $L_i$ is the smoothness constant of $f_i$, ${\color{red}c}$ is the maximal number of clients owning any feature ($1\leq {\color{red}c} \leq n$), and ${\color{blue}r}$ is the maximal number of features owned by any client ($1\leq {\color{blue}r} \leq d$).
Explicit Personalization and Local Training: Double Communication Acceleration in Federated Learning
Federated Learning is an evolving machine learning paradigm, in which multiple clients perform computations based on their individual private data, interspersed by communication with a remote server.
ELF: Federated Langevin Algorithms with Primal, Dual and Bidirectional Compression
Federated sampling algorithms have recently gained great popularity in the community of machine learning and statistics.
TAMUNA: Doubly Accelerated Distributed Optimization with Local Training, Compression, and Partial Participation
We propose TAMUNA, the first algorithm for distributed optimization that leveraged the two strategies of local training and compression jointly and allows for partial participation.
Federated Learning with Regularized Client Participation
Under this scheme, each client joins the learning process every $R$ communication rounds, which we refer to as a meta epoch.
High-Probability Bounds for Stochastic Optimization and Variational Inequalities: the Case of Unbounded Variance
During recent years the interest of optimization and machine learning communities in high-probability convergence of stochastic optimization methods has been growing.
Convergence of First-Order Algorithms for Meta-Learning with Moreau Envelopes
As a special case, our theory allows us to show the convergence of First-Order Model-Agnostic Meta-Learning (FO-MAML) to the vicinity of a solution of Moreau objective.
Can 5th Generation Local Training Methods Support Client Sampling? Yes!
The celebrated FedAvg algorithm of McMahan et al. (2017) is based on three components: client sampling (CS), data sampling (DS) and local training (LT).
GradSkip: Communication-Accelerated Local Gradient Methods with Better Computational Complexity
We study a class of distributed optimization algorithms that aim to alleviate high communication costs by allowing the clients to perform multiple local gradient-type training steps prior to communication.
Provably Doubly Accelerated Federated Learning: The First Theoretically Successful Combination of Local Training and Communication Compression
In federated learning, a large number of users are involved in a global learning task, in a collaborative way.
Improved Stein Variational Gradient Descent with Importance Weights
In the continuous time and infinite particles regime, the time for this flow to converge to the equilibrium distribution $\pi$, quantified by the Stein Fisher information, depends on $\rho_0$ and $\pi$ very weakly.
Minibatch Stochastic Three Points Method for Unconstrained Smooth Minimization
In this paper, we propose a new zero order optimization method called minibatch stochastic three points (MiSTP) method to solve an unconstrained minimization problem in a setting where only an approximation of the objective function evaluation is possible.
Personalized Federated Learning with Communication Compression
Recently, Hanzely and Richt\'{a}rik (2020) proposed a new formulation for training personalized FL models aimed at balancing the trade-off between the traditional global model and the local models that could be trained by individual devices using their private data only.
Adaptive Learning Rates for Faster Stochastic Gradient Methods
In this work, we propose new adaptive step size strategies that improve several stochastic gradient methods.
Variance Reduced ProxSkip: Algorithm, Theory and Application to Federated Learning
We study distributed optimization methods based on the {\em local training (LT)} paradigm: achieving communication efficiency by performing richer local gradient-based training on the clients before parameter averaging.
Communication Acceleration of Local Gradient Methods via an Accelerated Primal-Dual Algorithm with Inexact Prox
Inspired by a recent breakthrough of Mishchenko et al (2022), who for the first time showed that local gradient steps can lead to provable communication acceleration, we propose an alternative algorithm which obtains the same communication acceleration as their method (ProxSkip).
Shifted Compression Framework: Generalizations and Improvements
Communication is one of the key bottlenecks in the distributed training of large-scale machine learning models, and lossy compression of exchanged information, such as stochastic gradients or models, is one of the most effective instruments to alleviate this issue.
A Note on the Convergence of Mirrored Stein Variational Gradient Descent under $(L_0,L_1)-$Smoothness Condition
In this note, we establish a descent lemma for the population limit Mirrored Stein Variational Gradient Method~(MSVGD).
Federated Optimization Algorithms with Random Reshuffling and Gradient Compression
To reveal the true advantages of RR in the distributed learning with compression, we propose a new method called DIANA-RR that reduces the compression variance and has provably better convergence rates than existing counterparts with with-replacement sampling of stochastic gradients.
Distributed Newton-Type Methods with Communication Compression and Bernoulli Aggregation
Despite their high computation and communication costs, Newton-type methods remain an appealing option for distributed training due to their robustness against ill-conditioned convex problems.
Certified Robustness in Federated Learning
However, as in the single-node supervised learning setup, models trained in federated learning suffer from vulnerability to imperceptible input transformations known as adversarial attacks, questioning their deployment in security-related applications.
Sharper Rates and Flexible Framework for Nonconvex SGD with Client and Data Sampling
The optimal complexity of stochastic first-order methods in terms of the number of gradient evaluations of individual functions is $\mathcal{O}\left(n + n^{1/2}\varepsilon^{-1}\right)$, attained by the optimal SGD methods $\small\sf\color{green}{SPIDER}$(arXiv:1807. 01695) and $\small\sf\color{green}{PAGE}$(arXiv:2008. 10898), for example, where $\varepsilon$ is the error tolerance.
Federated Learning with a Sampling Algorithm under Isoperimetry
Federated learning uses a set of techniques to efficiently distribute the training of a machine learning algorithm across several devices, who own the training data.
Variance Reduction is an Antidote to Byzantines: Better Rates, Weaker Assumptions and Communication Compression as a Cherry on the Top
However, many fruitful directions, such as the usage of variance reduction for achieving robustness and communication compression for reducing communication costs, remain weakly explored in the field.
A Computation and Communication Efficient Method for Distributed Nonconvex Problems in the Partial Participation Setting
We present a new method that includes three key components of distributed optimization and federated learning: variance reduction of stochastic gradients, partial participation, and compressed communication.
EF-BV: A Unified Theory of Error Feedback and Variance Reduction Mechanisms for Biased and Unbiased Compression in Distributed Optimization
Our general approach works with a new, larger class of compressors, which has two parameters, the bias and the variance, and includes unbiased and biased compressors as particular cases.
Federated Random Reshuffling with Compression and Variance Reduction
Random Reshuffling (RR), which is a variant of Stochastic Gradient Descent (SGD) employing sampling without replacement, is an immensely popular method for training supervised machine learning models via empirical risk minimization.
FedShuffle: Recipes for Better Use of Local Work in Federated Learning
The practice of applying several local updates before aggregation across clients has been empirically shown to be a successful approach to overcoming the communication bottleneck in Federated Learning (FL).
ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!
The canonical approach to solving such problems is via the proximal gradient descent (ProxGD) algorithm, which is based on the evaluation of the gradient of $f$ and the prox operator of $\psi$ in each iteration.
FL_PyTorch: optimization research simulator for federated learning
Our system supports abstractions that provide researchers with a sufficient level of flexibility to experiment with existing and novel approaches to advance the state-of-the-art.
Optimal Algorithms for Decentralized Stochastic Variational Inequalities
Our algorithms are the best among the available literature not only in the decentralized stochastic case, but also in the decentralized deterministic and non-distributed stochastic cases.
3PC: Three Point Compressors for Communication-Efficient Distributed Training and a Better Theory for Lazy Aggregation
We propose and study a new class of gradient communication mechanisms for communication-efficient training -- three point compressors (3PC) -- as well as efficient distributed nonconvex optimization algorithms that can take advantage of them.
DASHA: Distributed Nonconvex Optimization with Communication Compression, Optimal Oracle Complexity, and No Client Synchronization
When the local functions at the nodes have a finite-sum or an expectation form, our new methods, DASHA-PAGE and DASHA-SYNC-MVR, improve the theoretical oracle and communication complexity of the previous state-of-the-art method MARINA by Gorbunov et al. (2020).
BEER: Fast $O(1/T)$ Rate for Decentralized Nonconvex Optimization with Communication Compression
Communication efficiency has been widely recognized as the bottleneck for large-scale decentralized machine learning applications in multi-agent or federated environments.
Server-Side Stepsizes and Sampling Without Replacement Provably Help in Federated Optimization
Together, our results on the advantage of large and small server-side stepsizes give a formal justification for the practice of adaptive server-side optimization in federated learning.
Accelerated Primal-Dual Gradient Method for Smooth and Convex-Concave Saddle-Point Problems with Bilinear Coupling
In this paper we study the convex-concave saddle-point problem $\min_x \max_y f(x) + y^T \mathbf{A} x - g(y)$, where $f(x)$ and $g(y)$ are smooth and convex functions.
Faster Rates for Compressed Federated Learning with Client-Variance Reduction
In the convex setting, COFIG converges within $O(\frac{(1+\omega)\sqrt{N}}{S\epsilon})$ communication rounds, which, to the best of our knowledge, is also the first convergence result for compression schemes that do not communicate with all the clients in each round.
FLIX: A Simple and Communication-Efficient Alternative to Local Methods in Federated Learning
A persistent problem in federated learning is that it is not clear what the optimization objective should be: the standard average risk minimization of supervised learning is inadequate in handling several major constraints specific to federated learning, such as communication adaptivity and personalization control.
Basis Matters: Better Communication-Efficient Second Order Methods for Federated Learning
Recent advances in distributed optimization have shown that Newton-type methods with proper communication compression mechanisms can guarantee fast local rates and low communication cost compared to first order methods.
Distributed Methods with Compressed Communication for Solving Variational Inequalities, with Theoretical Guarantees
Due to these considerations, it is important to equip existing methods with strategies that would allow to reduce the volume of transmitted information during training while obtaining a model of comparable quality.
Permutation Compressors for Provably Faster Distributed Nonconvex Optimization
In this paper we i) extend the theory of MARINA to support a much wider class of potentially {\em correlated} compressors, extending the reach of the method beyond the classical independent compressors setting, ii) show that a new quantity, for which we coin the name {\em Hessian variance}, allows us to significantly refine the original analysis of MARINA without any additional assumptions, and iii) identify a special class of correlated compressors based on the idea of {\em random permutations}, for which we coin the term Perm$K$, the use of which leads to $O(\sqrt{n})$ (resp.
EF21 with Bells & Whistles: Practical Algorithmic Extensions of Modern Error Feedback
First proposed by Seide (2014) as a heuristic, error feedback (EF) is a very popular mechanism for enforcing convergence of distributed gradient-based optimization methods enhanced with communication compression strategies based on the application of contractive compression operators.
ZeroSARAH: Efficient Nonconvex Finite-Sum Optimization with Zero Full Gradient Computations
Avoiding any full gradient computations (which are time-consuming steps) is important in many applications as the number of data samples $n$ usually is very large.
Doubly Adaptive Scaled Algorithm for Machine Learning Using Second-Order Information
We present a novel adaptive optimization algorithm for large-scale machine learning problems.
FedPAGE: A Fast Local Stochastic Gradient Method for Communication-Efficient Federated Learning
We propose a new federated learning algorithm, FedPAGE, able to further reduce the communication complexity by utilizing the recent optimal PAGE method (Li et al., 2021) instead of plain SGD in FedAvg.
CANITA: Faster Rates for Distributed Convex Optimization with Communication Compression
Due to the high communication cost in distributed and federated learning, methods relying on compressed communication are becoming increasingly popular.
EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedback
However, all existing analyses either i) apply to the single node setting only, ii) rely on very strong and often unreasonable assumptions, such global boundedness of the gradients, or iterate-dependent assumptions that cannot be checked a-priori and may not hold in practice, or iii) circumvent these issues via the introduction of additional unbiased compressors, which increase the communication cost.
Theoretically Better and Numerically Faster Distributed Optimization with Smoothness-Aware Quantization Techniques
To address the high communication costs of distributed machine learning, a large body of work has been devoted in recent years to designing various compression strategies, such as sparsification and quantization, and optimization algorithms capable of using them.
A Convergence Theory for SVGD in the Population Limit under Talagrand's Inequality T1
We first establish the convergence of the algorithm.
MURANA: A Generic Framework for Stochastic Variance-Reduced Optimization
We propose a generic variance-reduced algorithm, which we call MUltiple RANdomized Algorithm (MURANA), for minimizing a sum of several smooth functions plus a regularizer, in a sequential or distributed manner.
FedNL: Making Newton-Type Methods Applicable to Federated Learning
In contrast to the aforementioned work, FedNL employs a different Hessian learning technique which i) enhances privacy as it does not rely on the training data to be revealed to the coordinating server, ii) makes it applicable beyond generalized linear models, and iii) provably works with general contractive compression operators for compressing the local Hessians, such as Top-$K$ or Rank-$R$, which are vastly superior in practice.
Random Reshuffling with Variance Reduction: New Analysis and Better Rates
One of the tricks that works so well in practice that it is used as default in virtually all widely used machine learning software is {\em random reshuffling (RR)}.
ZeroSARAH: Efficient Nonconvex Finite-Sum Optimization with Zero Full Gradient Computation
Avoiding any full gradient computations (which are time-consuming steps) is important in many applications as the number of data samples $n$ usually is very large.
Hyperparameter Transfer Learning with Adaptive Complexity
Bayesian optimization (BO) is a sample efficient approach to automatically tune the hyperparameters of machine learning models.
An Optimal Algorithm for Strongly Convex Minimization under Affine Constraints
Optimization problems under affine constraints appear in various areas of machine learning.
Optimization and Control
AI-SARAH: Adaptive and Implicit Stochastic Recursive Gradient Methods
We present AI-SARAH, a practical variant of SARAH.
ADOM: Accelerated Decentralized Optimization Method for Time-Varying Networks
We propose ADOM - an accelerated method for smooth and strongly convex decentralized optimization over time-varying networks.
IntSGD: Adaptive Floatless Compression of Stochastic Gradients
We propose a family of adaptive integer compression operators for distributed Stochastic Gradient Descent (SGD) that do not communicate a single float.
MARINA: Faster Non-Convex Distributed Learning with Compression
Unlike virtually all competing distributed first-order methods, including DIANA, ours is based on a carefully designed biased gradient estimator, which is the key to its superior theoretical and practical performance.
Distributed Second Order Methods with Fast Rates and Compressed Communication
Finally, we develop a globalization strategy using cubic regularization which leads to our next method, CUBIC-NEWTON-LEARN, for which we prove global sublinear and linear convergence rates, and a fast superlinear rate.
Smoothness Matrices Beat Smoothness Constants: Better Communication Compression Techniques for Distributed Optimization
In order to further alleviate the communication burden inherent in distributed optimization, we propose a novel communication sparsification strategy that can take full advantage of the smoothness matrices associated with local losses.
Proximal and Federated Random Reshuffling
Random Reshuffling (RR), also known as Stochastic Gradient Descent (SGD) without replacement, is a popular and theoretically grounded method for finite-sum minimization.
Local SGD: Unified Theory and New Efficient Methods
We present a unified framework for analyzing local SGD methods in the convex and strongly convex regimes for distributed/federated training of supervised machine learning models.
Linearly Converging Error Compensated SGD
Moreover, using our general scheme, we develop new variants of SGD that combine variance reduction or arbitrary sampling with error feedback and quantization and derive the convergence rates for these methods beating the state-of-the-art results.
Optimal Gradient Compression for Distributed and Federated Learning
In the average-case analysis, we design a simple compression operator, Spherical Compression, which naturally achieves the lower bound.
Lower Bounds and Optimal Algorithms for Personalized Federated Learning
Our first contribution is establishing the first lower bounds for this formulation, for both the communication complexity and the local oracle complexity.
Distributed Proximal Splitting Algorithms with Rates and Acceleration
We analyze several generic proximal splitting algorithms well suited for large-scale convex nonsmooth optimization.
PAGE: A Simple and Optimal Probabilistic Gradient Estimator for Nonconvex Optimization
Then, we show that PAGE obtains the optimal convergence results $O(n+\frac{\sqrt{n}}{\epsilon^2})$ (finite-sum) and $O(b+\frac{\sqrt{b}}{\epsilon^2})$ (online) matching our lower bounds for both nonconvex finite-sum and online problems.
Optimal and Practical Algorithms for Smooth and Strongly Convex Decentralized Optimization
We propose two new algorithms for this decentralized optimization problem and equip them with complexity guarantees.
Unified Analysis of Stochastic Gradient Methods for Composite Convex and Smooth Optimization
We showcase this by obtaining a simple formula for the optimal minibatch size of two variance reduced methods (\textit{L-SVRG} and \textit{SAGA}).
A Better Alternative to Error Feedback for Communication-Efficient Distributed Learning
EF remains the only known technique that can deal with the error induced by contractive compressors which are not unbiased, such as Top-$K$.
Primal Dual Interpretation of the Proximal Stochastic Gradient Langevin Algorithm
In the second part of this paper, we use the duality gap arising from the first part to study the complexity of the Proximal Stochastic Gradient Langevin Algorithm (PSGLA), which can be seen as a generalization of the Projected Langevin Algorithm.
A Unified Analysis of Stochastic Gradient Methods for Nonconvex Federated Optimization
We provide a single convergence analysis for all methods that satisfy the proposed unified assumption, thereby offering a unified understanding of SGD variants in the nonconvex regime instead of relying on dedicated analyses of each variant.
Random Reshuffling: Simple Analysis with Vast Improvements
from $\kappa$ to $\sqrt{\kappa}$) and, in addition, show that RR has a different type of variance.
From Local SGD to Local Fixed-Point Methods for Federated Learning
Most algorithms for solving optimization problems or finding saddle points of convex-concave functions are fixed-point algorithms.
Dualize, Split, Randomize: Toward Fast Nonsmooth Optimization Algorithms
We consider minimizing the sum of three convex functions, where the first one F is smooth, the second one is nonsmooth and proximable and the third one is the composition of a nonsmooth proximable function with a linear operator L. This template problem has many applications, for instance, in image processing and machine learning.
On Biased Compression for Distributed Learning
In the last few years, various communication compression techniques have emerged as an indispensable tool helping to alleviate the communication bottleneck in distributed learning.
Acceleration for Compressed Gradient Descent in Distributed and Federated Optimization
Due to the high communication cost in distributed and federated learning problems, methods relying on compression of communicated messages are becoming increasingly popular.
Stochastic Subspace Cubic Newton Method
In this paper, we propose a new randomized second-order optimization algorithm---Stochastic Subspace Cubic Newton (SSCN)---for minimizing a high dimensional convex function $f$.
Uncertainty Principle for Communication Compression in Distributed and Federated Learning and the Search for an Optimal Compressor
In designing a compression method, one aims to communicate as few bits as possible, which minimizes the cost per communication round, while at the same time attempting to impart as little distortion (variance) to the communicated messages as possible, which minimizes the adverse effect of the compression on the overall number of communication rounds.
Adaptivity of Stochastic Gradient Methods for Nonconvex Optimization
Adaptivity is an important yet under-studied property in modern optimization theory.
Federated Learning of a Mixture of Global and Local Models
We propose a new optimization formulation for training federated learning models.
Better Theory for SGD in the Nonconvex World
Moreover, we perform our analysis in a framework which allows for a detailed study of the effects of a wide array of sampling strategies and minibatch sizes for finite-sum optimization problems.
Distributed Fixed Point Methods with Compressed Iterates
We propose basic and natural assumptions under which iterative optimization methods with compressed iterates can be analyzed.
Stochastic Newton and Cubic Newton Methods with Simple Local Linear-Quadratic Rates
We present two new remarkably simple stochastic second-order methods for minimizing the average of a very large number of sufficiently smooth and strongly convex functions.
Learning to Optimize via Dual space Preconditioning
Preconditioning an minimization algorithm improve its convergence and can lead to a minimizer in one iteration in some extreme cases.
On Stochastic Sign Descent Methods
Various gradient compression schemes have been proposed to mitigate the communication cost in distributed training of large scale machine learning models.
Tighter Theory for Local SGD on Identical and Heterogeneous Data
We provide a new analysis of local SGD, removing unnecessary assumptions and elaborating on the difference between two data regimes: identical and heterogeneous.
Gradient Descent with Compressed Iterates
We propose and analyze a new type of stochastic first order method: gradient descent with compressed iterates (GDCI).
First Analysis of Local GD on Heterogeneous Data
We provide the first convergence analysis of local gradient descent for minimizing the average of smooth and convex but otherwise arbitrary functions.
Stochastic Convolutional Sparse Coding
In this work, we propose a novel stochastic spatial-domain solver, in which a randomized subsampling strategy is introduced during the learning sparse codes.
Stochastic Proximal Langevin Algorithm: Potential Splitting and Nonasymptotic Rates
We propose a new algorithm---Stochastic Proximal Langevin Algorithm (SPLA)---for sampling from a log concave distribution.
Direct Nonlinear Acceleration
In contrast to RNA which computes extrapolation coefficients by (approximately) setting the gradient of the objective function to zero at the extrapolated point, we propose a more direct approach, which we call direct nonlinear acceleration (DNA).
Revisiting Stochastic Extragradient
We fix a fundamental issue in the stochastic extragradient method by providing a new sampling strategy that is motivated by approximating implicit updates.
One Method to Rule Them All: Variance Reduction for Data, Parameters and Many New Methods
We propose a remarkably general variance-reduced method suitable for solving regularized empirical risk minimization problems with either a large number of training examples, or a large model dimension, or both.
A Unified Theory of SGD: Variance Reduction, Sampling, Quantization and Coordinate Descent
In this paper we introduce a unified analysis of a large family of variants of proximal stochastic gradient descent ({\tt SGD}) which so far have required different intuitions, convergence analyses, have different applications, and which have been developed separately in various communities.
Best Pair Formulation & Accelerated Scheme for Non-convex Principal Component Pursuit
The best pair problem aims to find a pair of points that minimize the distance between two disjoint sets.
Revisiting Randomized Gossip Algorithms: General Framework, Convergence Rates and Novel Block and Accelerated Protocols
In this work we present a new framework for the analysis and design of randomized gossip algorithms for solving the average consensus problem.
Convergence Analysis of Inexact Randomized Iterative Methods
We relax this requirement by allowing for the sub-problem to be solved inexactly.
Scaling Distributed Machine Learning with In-Network Aggregation
Training machine learning models in parallel is an increasingly important workload.
Quasi-Newton Methods for Machine Learning: Forget the Past, Just Sample
We present two sampled quasi-Newton methods (sampled LBFGS and sampled LSR1) for solving empirical risk minimization problems that arise in machine learning.
A Privacy Preserving Randomized Gossip Algorithm via Controlled Noise Insertion
In this work we present a randomized gossip algorithm for solving the average consensus problem while at the same time protecting the information about the initial private values stored at the nodes.
99% of Distributed Optimization is a Waste of Time: The Issue and How to Fix it
We propose a fix based on a new update-sparsification method we develop in this work, which we suggest be used on top of existing methods.
Distributed Learning with Compressed Gradient Differences
Our analysis of block-quantization and differences between $\ell_2$ and $\ell_{\infty}$ quantization closes the gaps in theory and practice.
SAGA with Arbitrary Sampling
We study the problem of minimizing the average of a very large number of smooth functions, which is of key importance in training supervised learning models.
New Convergence Aspects of Stochastic Gradient Algorithms
We show the convergence of SGD for strongly convex objective function without using bounded gradient assumption when $\{\eta_t\}$ is a diminishing sequence and $\sum_{t=0}^\infty \eta_t \rightarrow \infty$.
Provably Accelerated Randomized Gossip Algorithms
In this work we present novel provably accelerated gossip algorithms for solving the average consensus problem.
Accelerated Gossip via Stochastic Heavy Ball Method
In this paper we show how the stochastic heavy ball method (SHB) -- a popular method for solving stochastic convex and non-convex optimization problems --operates as a randomized gossip algorithm.
A Nonconvex Projection Method for Robust PCA
Robust principal component analysis (RPCA) is a well-studied problem with the goal of decomposing a matrix into the sum of low-rank and sparse components.
SGD and Hogwild! Convergence Without the Bounded Gradients Assumption
In (Bottou et al., 2016), a new analysis of convergence of SGD is performed under the assumption that stochastic gradients are bounded with respect to the true gradient norm.
Momentum and Stochastic Momentum for Stochastic Gradient, Newton, Proximal Point and Subspace Descent Methods
We prove linear convergence of several stochastic methods with stochastic momentum, and show that in some sparse data regimes and for sufficiently small momentum parameters, these methods enjoy better overall complexity than methods with deterministic momentum.
Linearly convergent stochastic heavy ball method for minimizing generalization error
In this work we establish the first linear convergence result for the stochastic heavy ball method.
A Batch-Incremental Video Background Estimation Model using Weighted Low-Rank Approximation of Matrices
Principal component pursuit (PCP) is a state-of-the-art approach for background estimation problems.
Privacy Preserving Randomized Gossip Algorithms
In this work we present three different randomized gossip algorithms for solving the average consensus problem while at the same time protecting the information about the initial private values stored at the nodes.
Optimization and Control
Stochastic Primal-Dual Hybrid Gradient Algorithm with Arbitrary Sampling and Imaging Applications
We propose a stochastic extension of the primal-dual hybrid gradient algorithm studied by Chambolle and Pock in 2011 to solve saddle point problems that are separable in the dual variable.
Stochastic Reformulations of Linear Systems: Algorithms and Convergence Theory
We develop a family of reformulations of an arbitrary consistent linear system into a stochastic problem.
Randomized Distributed Mean Estimation: Accuracy vs Communication
We consider the problem of estimating the arithmetic average of a finite collection of real vectors stored in a distributed fashion across several compute nodes subject to a communication budget constraint.
Federated Learning: Strategies for Improving Communication Efficiency
We consider learning algorithms for this setting where on each round, each client independently computes an update to the current model based on its local data, and communicates this update to a central server, where the client-side updates are aggregated to compute a new global model.
Federated Optimization: Distributed Machine Learning for On-Device Intelligence
We refer to this setting as Federated Optimization.
AIDE: Fast and Communication Efficient Distributed Optimization
It is well known that DANE algorithm does not match the communication complexity lower bounds.
Importance Sampling for Minibatches
Minibatching is a very well studied and highly popular technique in supervised learning, used by practitioners due to its ability to accelerate training through better utilization of parallel processing power and reduction of stochastic variance.
Even Faster Accelerated Coordinate Descent Using Non-Uniform Sampling
Accelerated coordinate descent is widely used in optimization due to its cheap per-iteration cost and scalability to large-scale problems.
Distributed Optimization with Arbitrary Local Solvers
To this end, we present a framework for distributed optimization that both allows the flexibility of arbitrary solvers to be used on each (single) machine locally, and yet maintains competitive performance against other state-of-the-art special-purpose distributed methods.
Distributed Mini-Batch SDCA
We present an improved analysis of mini-batched stochastic dual coordinate ascent for regularized empirical loss minimization (i. e. SVM and SVM-type objectives).
Primal Method for ERM with Flexible Mini-batching Schemes and Non-convex Losses
For convex loss functions, our complexity results match those of QUARTZ, which is a primal-dual method also allowing for arbitrary mini-batching schemes.
Mini-Batch Semi-Stochastic Gradient Descent in the Proximal Setting
Our method first performs a deterministic step (computation of the gradient of the objective function at the starting point), followed by a large number of stochastic steps.
Stochastic Dual Coordinate Ascent with Adaptive Probabilities
This paper introduces AdaSDCA: an adaptive variant of stochastic dual coordinate ascent (SDCA) for solving the regularized empirical risk minimization problems.
Adding vs. Averaging in Distributed Primal-Dual Optimization
Distributed optimization methods for large-scale machine learning suffer from a communication bottleneck.
SDNA: Stochastic Dual Newton Ascent for Empirical Risk Minimization
We propose a new algorithm for minimizing regularized empirical loss: Stochastic Dual Newton Ascent (SDNA).
Coordinate Descent with Arbitrary Sampling II: Expected Separable Overapproximation
The design and complexity analysis of randomized coordinate descent methods, and in particular of variants which update a random subset (sampling) of coordinates in each iteration, depends on the notion of expected separable overapproximation (ESO).
Coordinate Descent with Arbitrary Sampling I: Algorithms and Complexity
ALPHA is a remarkably flexible algorithm: in special cases, it reduces to deterministic and randomized methods such as gradient descent, coordinate descent, parallel coordinate descent and distributed coordinate descent -- both in nonaccelerated and accelerated variants.
Randomized Dual Coordinate Ascent with Arbitrary Sampling
The distributed variant of Quartz is the first distributed SDCA-like method with an analysis for non-separable data.
mS2GD: Mini-Batch Semi-Stochastic Gradient Descent in the Proximal Setting
Our method first performs a deterministic step (computation of the gradient of the objective function at the starting point), followed by a large number of stochastic steps.
Fast Distributed Coordinate Descent for Non-Strongly Convex Losses
We propose an efficient distributed randomized coordinate descent method for minimizing regularized non-strongly convex loss functions.
Accelerated, Parallel and Proximal Coordinate Descent
In the special case when the number of processors is equal to the number of coordinates, the method converges at the rate $2\bar{\omega}\bar{L} R^2/(k+1)^2 $, where $k$ is the iteration counter, $\bar{\omega}$ is an average degree of separability of the loss function, $\bar{L}$ is the average of Lipschitz constants associated with the coordinates and individual functions in the sum, and $R$ is the distance of the initial point from the minimizer.
Semi-Stochastic Gradient Descent Methods
The total work needed for the method to output an $\varepsilon$-accurate solution in expectation, measured in the number of passes over data, or equivalently, in units equivalent to the computation of a single gradient of the loss, is $O((\kappa/n)\log(1/\varepsilon))$, where $\kappa$ is the condition number.
TOP-SPIN: TOPic discovery via Sparse Principal component INterference
Our approach attacks the maximization problem in sparse PCA directly and is scalable to high-dimensional data.
On Optimal Probabilities in Stochastic Coordinate Descent Methods
We propose and analyze a new parallel coordinate descent method---`NSync---in which at each iteration a random subset of coordinates is updated, in parallel, allowing for the subsets to be chosen non-uniformly.
Distributed Coordinate Descent Method for Learning with Big Data
In this paper we develop and analyze Hydra: HYbriD cooRdinAte descent method for solving loss minimization problems with big data.
Smooth minimization of nonsmooth functions with parallel coordinate descent methods
We study the performance of a family of randomized parallel coordinate descent methods for minimizing the sum of a nonsmooth and separable convex functions.
Inexact Coordinate Descent: Complexity and Preconditioning
In this paper we consider the problem of minimizing a convex function using a randomized block coordinate descent method.
Alternating Maximization: Unifying Framework for 8 Sparse PCA Formulations and Efficient Parallel Codes
Given a multivariate data set, sparse principal component analysis (SPCA) aims to extract several linear combinations of the variables that together explain the variance in the data as much as possible, while controlling the number of nonzero loadings in these combinations.
Parallel Coordinate Descent Methods for Big Data Optimization
In this work we show that randomized (block) coordinate descent methods can be accelerated by parallelization when applied to the problem of minimizing the sum of a partially separable smooth convex function and a simple separable convex function.
