Search Results for author:
Found 4 papers, 1 papers with code
Ultra-Low Precision 4-bit Training of Deep Neural Networks
In this paper, we propose a number of novel techniques and numerical representation formats that enable, for the very first time, the precision of training systems to be aggressively scaled from 8-bits to 4-bits.
Accumulation Bit-Width Scaling For Ultra-Low Precision Training Of Deep Networks
Observing that a bad choice for accumulation precision results in loss of information that manifests itself as a reduction in variance in an ensemble of partial sums, we derive a set of equations that relate this variance to the length of accumulation and the minimum number of bits needed for accumulation.
AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training
Highly distributed training of Deep Neural Networks (DNNs) on future compute platforms (offering 100 of TeraOps/s of computational capacity) is expected to be severely communication constrained.
Deep Learning with Limited Numerical Precision
Training of large-scale deep neural networks is often constrained by the available computational resources.
