no code implementations • 18 Apr 2022 • Zhong-Jing Chen, Eduin E. Hernandez, Yu-Chih Huang, Stefano Rini
Namely: (i) gradient quantization through floating-point conversion, (ii) lossless compression of the quantized gradient, and (iii) quantization error correction.
1 code implementation • 17 Mar 2022 • Zhong-Jing Chen, Eduin E. Hernandez, Yu-Chih Huang, Stefano Rini
In this paper, we introduce a novel algorithm, $\mathsf{CO}_3$, for communication-efficiency distributed Deep Neural Network (DNN) training.
1 code implementation • 15 Nov 2021 • Zhong-Jing Chen, Eduin E. Hernandez, Yu-Chih Huang, Stefano Rini
In this paper we argue that, for some networks of practical interest, the gradient entries can be well modelled as having a generalized normal (GenNorm) distribution.
1 code implementation • 18 Oct 2021 • Eduin E. Hernandez, Stefano Rini, Tolga M. Duman
In order to correct for the inherent bias in this approximation, the algorithm retains in memory an accumulation of the outer products that are not used in the approximation.