Search Results for author: Jiaao He

Found 3 papers, 1 papers with code

FastMoE: A Fast Mixture-of-Expert Training System

3 code implementations24 Mar 2021 Jiaao He, Jiezhong Qiu, Aohan Zeng, Zhilin Yang, Jidong Zhai, Jie Tang

However, training trillion-scale MoE requires algorithm and system co-design for a well-tuned high performance distributed training system.

Language Modelling

Heterogeneity-Aware Asynchronous Decentralized Training

no code implementations17 Sep 2019 Qinyi Luo, Jiaao He, Youwei Zhuo, Xuehai Qian

Is it possible to get the best of both worlds - designing a distributed training method that has both high performance as All-Reduce in homogeneous environment and good heterogeneity tolerance as AD-PSGD?


Cannot find the paper you are looking for? You can Submit a new open access paper.