Search Results for author:
Found 17 papers, 3 papers with code
Asynchronous SGD on Graphs: a Unified Framework for Asynchronous Decentralized and Federated Optimization
Decentralized and asynchronous communications are two popular techniques to speedup communication complexity of distributed machine learning, by respectively removing the dependency over a central orchestrator and the need for synchronization.
On Convergence of Incremental Gradient for Non-Convex Smooth Functions
In machine learning and neural network optimization, algorithms like incremental gradient, and shuffle SGD are popular due to minimizing the number of cache misses and good practical convergence behavior.
Revisiting Gradient Clipping: Stochastic bias and tight convergence guarantees
In particular, we show that (i) for deterministic gradient descent, the clipping threshold only affects the higher-order terms of convergence, (ii) in the stochastic setting convergence to the true optimum cannot be guaranteed under the standard noise assumption, even under arbitrary small step-sizes.
Gradient Descent with Linearly Correlated Noise: Theory and Applications to Differential Privacy
We propose a simplified setting that distills key facets of these methods and isolates the impact of linearly correlated noise.
Decentralized Gradient Tracking with Local Steps
Gradient tracking (GT) is an algorithm designed for solving decentralized optimization problems over a network (such as training a machine learning model).
Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learning
In this work (i) we obtain a tighter convergence rate of $\mathcal{O}\!\left(\sigma^2\epsilon^{-2}+ \sqrt{\tau_{\max}\tau_{avg}}\epsilon^{-1}\right)$ without any change in the algorithm where $\tau_{avg}$ is the average delay, which can be significantly smaller than $\tau_{\max}$.
Data-heterogeneity-aware Mixing for Decentralized Learning
Decentralized learning provides an effective framework to train machine learning models with data distributed over arbitrary communication graphs.
An Improved Analysis of Gradient Tracking for Decentralized Machine Learning
We consider decentralized machine learning over a network where the training data is distributed across $n$ agents, each of which can compute stochastic model updates on their local data.
RelaySum for Decentralized Deep Learning on Heterogeneous Data
A key challenge, primarily in decentralized deep learning, remains the handling of differences between the workers' local data distributions.
Decentralized Local Stochastic Extra-Gradient for Variational Inequalities
We extend the stochastic extragradient method to this very general setting and theoretically analyze its convergence rate in the strongly-monotone, monotone, and non-monotone (when a Minty solution exists) settings.
Consensus Control for Decentralized Deep Learning
Decentralized training of deep learning models enables on-device learning over networks, as well as efficient scaling to large compute clusters.
On the Effect of Consensus in Decentralized Deep Learning
Decentralized training of deep learning models enables on-device learning over networks, as well as efficient scaling to large compute clusters.
A Linearly Convergent Algorithm for Decentralized Optimization: Sending Less Bits for Free!
Decentralized optimization methods enable on-device training of machine learning models without a central coordinator.
A Unified Theory of Decentralized SGD with Changing Topology and Local Updates
Decentralized stochastic optimization methods have gained a lot of attention recently, mainly because of their cheap per iteration cost, data locality, and their communication-efficiency.
Decentralized Deep Learning with Arbitrary Communication Compression
Decentralized training of deep learning models is a key element for enabling data privacy and on-device learning over networks, as well as for efficient scaling to large compute clusters.
Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
We (i) propose a novel gossip-based stochastic gradient descent algorithm, CHOCO-SGD, that converges at rate $\mathcal{O}\left(1/(nT) + 1/(T \delta^2 \omega)^2\right)$ for strongly convex objectives, where $T$ denotes the number of iterations and $\delta$ the eigengap of the connectivity matrix.
Efficient Greedy Coordinate Descent for Composite Problems
For these problems we provide (i) the first linear rates of convergence independent of $n$, and show that our greedy update rule provides speedups similar to those obtained in the smooth case.
