Search Results for author:
Found 4 papers, 2 papers with code
Resource-Efficient Federated Learning
Federated Learning (FL) enables distributed training by learners using local data, thereby enhancing privacy and reducing communication.
Rethinking gradient sparsification as total error minimization
Unlike with Top-$k$ sparsifier, we show that hard-threshold has the same asymptotic convergence and linear speedup property as SGD in the convex case and has no impact on the data-heterogeneity in the non-convex case.
On the Discrepancy between the Theoretical Analysis and Practical Implementations of Compressed Communication for Distributed Deep Learning
Compressed communication, in the form of sparsification or quantization of stochastic gradients, is employed to reduce communication costs in distributed data-parallel training of deep neural networks.
Natural Compression for Distributed Deep Learning
Our technique is applied individually to all entries of the to-be-compressed update vector and works by randomized rounding to the nearest (negative or positive) power of two, which can be computed in a "natural" way by ignoring the mantissa.
