Search Results for author:
Found 3 papers, 1 papers with code
Heterogeneity-Aware Asynchronous Decentralized Training
Is it possible to get the best of both worlds - designing a distributed training method that has both high performance as All-Reduce in homogeneous environment and good heterogeneity tolerance as AD-PSGD?
FastMoE: A Fast Mixture-of-Expert Training System
However, training trillion-scale MoE requires algorithm and system co-design for a well-tuned high performance distributed training system.
A Roadmap for Big Model
With the rapid development of deep learning, training Big Models (BMs) for multiple downstream tasks becomes a popular paradigm.
Cannot find the paper you are looking for? You can
Submit a new open access paper.
