Search Results for author:
Found 11 papers, 2 papers with code
Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularization
We show that regularizing both the inputs and outputs is crucial for preventing a model's "migrating" the difficulty in input quantization to the weights, which makes post-training quantization (PTQ) of weights more difficult.
COMQ: A Backpropagation-Free Algorithm for Post-Training Quantization
Within a fixed layer, COMQ treats all the scaling factor(s) and bit-codes as the variables of the reconstruction error.
4-bit Quantization of LSTM-based Speech Recognition Models
We investigate the impact of aggressive low-precision representations of weights and activations in two families of large LSTM-based architectures for Automatic Speech Recognition (ASR): hybrid Deep Bidirectional LSTM - Hidden Markov Models (DBLSTM-HMMs) and Recurrent Neural Network - Transducers (RNN-Ts).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
ScaleCom: Scalable Sparsified Gradient Compression for Communication-Efficient Distributed Training
Large-scale distributed training of Deep Neural Networks (DNNs) on state-of-the-art platforms is expected to be severely communication constrained.
Improved Techniques for Quantizing Deep Networks with Adaptive Bit-Widths
Second, to effectively transfer knowledge, we develop a dynamic block swapping method by randomly replacing the blocks in the lower-precision student network with the corresponding blocks in the higher-precision teacher network.
A Comprehensive Survey on Hardware-Aware Neural Architecture Search
Arguably their most significant impact has been in image classification and object detection tasks where the state of the art results have been obtained.
Hardware Aware Neural Architecture Search
Image Classification
+3
Ultra-Low Precision 4-bit Training of Deep Neural Networks
In this paper, we propose a number of novel techniques and numerical representation formats that enable, for the very first time, the precision of training systems to be aggressively scaled from 8-bits to 4-bits.
Hybrid 8-bit Floating Point (HFP8) Training and Inference for Deep Neural Networks
Reducing the numerical precision of data and computation is extremely effective in accelerating deep learning training workloads.
The Sooner The Better: Investigating Structure of Early Winning Lottery Tickets
In the case of ResNet50 on ImageNet, this comes to the winning ticket of 75:02% Top-1 accuracy at 80% pruning rate in only 22% of the total epochs for iterative pruning.
Accumulation Bit-Width Scaling For Ultra-Low Precision Training Of Deep Networks
Observing that a bad choice for accumulation precision results in loss of information that manifests itself as a reduction in variance in an ensemble of partial sums, we derive a set of equations that relate this variance to the length of accumulation and the minimum number of bits needed for accumulation.
Training Deep Neural Networks with 8-bit Floating Point Numbers
The state-of-the-art hardware platforms for training Deep Neural Networks (DNNs) are moving from traditional single precision (32-bit) computations towards 16 bits of precision -- in large part due to the high energy efficiency and smaller bit storage associated with using reduced-precision representations.
