Search Results for author:
Found 15 papers, 7 papers with code
Practical Insights into Knowledge Distillation for Pre-Trained Models
This research investigates the enhancement of knowledge distillation (KD) processes in pre-trained models, an emerging field in knowledge transfer with significant implications for distributed training and federated learning environments.
Flashback: Understanding and Mitigating Forgetting in Federated Learning
In Federated Learning (FL), forgetting, or the loss of knowledge across rounds, hampers algorithm convergence, particularly in the presence of severe data heterogeneity among clients.
Kimad: Adaptive Gradient Compression with Bandwidth Awareness
In distributed training, communication often emerges as a bottleneck.
Global-QSGD: Practical Floatless Quantization for Distributed Learning with Theoretical Guarantees
To obtain theoretical guarantees, we generalize the notion of standard unbiased compression operators to incorporate Global-QSGD.
FilFL: Client Filtering for Optimized Client Participation in Federated Learning
Federated learning is an emerging machine learning paradigm that enables clients to train collaboratively without exchanging local data.
Resource-Efficient Federated Learning
Federated Learning (FL) enables distributed training by learners using local data, thereby enhancing privacy and reducing communication.
Rethinking gradient sparsification as total error minimization
Unlike with Top-$k$ sparsifier, we show that hard-threshold has the same asymptotic convergence and linear speedup property as SGD in the convex case and has no impact on the data-heterogeneity in the non-convex case.
AutoLRS: Automatic Learning-Rate Schedule by Bayesian Optimization on the Fly
This mutual-training process between BO and the loss-prediction model allows us to limit the training steps invested in the BO search.
On the Impact of Device and Behavioral Heterogeneity in Federated Learning
Federated learning (FL) is becoming a popular paradigm for collaborative learning over distributed, private datasets owned by non-trusting entities.
An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systems
Many proposals exploit the compressibility of the gradients and propose lossy compression techniques to speed up the communication stage of distributed training.
On the Discrepancy between the Theoretical Analysis and Practical Implementations of Compressed Communication for Distributed Deep Learning
Compressed communication, in the form of sparsification or quantization of stochastic gradients, is employed to reduce communication costs in distributed data-parallel training of deep neural networks.
Direct Nonlinear Acceleration
In contrast to RNA which computes extrapolation coefficients by (approximately) setting the gradient of the objective function to zero at the extrapolated point, we propose a more direct approach, which we call direct nonlinear acceleration (DNA).
Natural Compression for Distributed Deep Learning
Our technique is applied individually to all entries of the to-be-compressed update vector and works by randomized rounding to the nearest (negative or positive) power of two, which can be computed in a "natural" way by ignoring the mantissa.
Scaling Distributed Machine Learning with In-Network Aggregation
Training machine learning models in parallel is an increasingly important workload.
Renaissance: A Self-Stabilizing Distributed SDN Control Plane using In-band Communications
By introducing programmability, automated verification, and innovative debugging tools, Software-Defined Networks (SDNs) are poised to meet the increasingly stringent dependability requirements of today's communication networks.
Networking and Internet Architecture Distributed, Parallel, and Cluster Computing Data Structures and Algorithms
