Search Results for author:
Found 25 papers, 8 papers with code
GPTVQ: The Blessing of Dimensionality for LLM Quantization
In this work we show that the size versus accuracy trade-off of neural network quantization can be significantly improved by increasing the quantization dimensionality.
The LLM Surgeon
Experimentally, our method can prune rows and columns from a range of OPT models and Llamav2-7B by 20%-30%, with a negligible loss in performance, and achieve state-of-the-art results in unstructured and semi-structured pruning of large language models.
MobileNVC: Real-time 1080p Neural Video Compression on a Mobile Device
This work presents the first neural video codec that decodes 1080p YUV420 video in real time on a mobile device.
Softmax Bias Correction for Quantized Generative Models
Post-training quantization (PTQ) is the go-to compression technique for large generative models, such as stable diffusion or large language models.
ResQ: Residual Quantization for Video Perception
Furthermore, we extend our model to dynamically adjust the bit-width proportional to the amount of changes in the video.
QBitOpt: Fast and Accurate Bitwidth Reallocation during Training
By combining fast-to-compute sensitivities with efficient solvers during QAT, QBitOpt can produce mixed-precision networks with high task performance guaranteed to satisfy strict resource constraints.
Pruning vs Quantization: Which is Better?
We provide an extensive comparison between the two techniques for compressing deep neural networks.
FP8 versus INT8 for efficient deep learning inference
We theoretically show the difference between the INT and FP formats for neural networks and present a plethora of post-training quantization and quantization-aware-training results to show how this theory translates to practice.
A Practical Mixed Precision Algorithm for Post-Training Quantization
We experimentally validate our proposed method on several computer vision tasks, natural language processing tasks and many different networks, and show that we can find mixed precision networks that provide a better trade-off between accuracy and efficiency than their homogeneous bit-width equivalents.
Quadapter: Adapter for GPT-2 Quantization
In that case, it is observed that quantization-aware training overfits the model to the fine-tuning data.
FP8 Quantization: The Power of the Exponent
We detail the choices that can be made for the FP8 format, including the important choice of the number of bits for the mantissa and exponent, and show analytically in which settings these choices give better performance.
Quantized Sparse Weight Decomposition for Neural Network Compression
In this paper, we introduce a novel method of neural network weight compression.
Quantization Robust Federated Learning for Efficient Inference on Heterogeneous Devices
We perform extensive experiments on standard FL benchmarks to evaluate our proposed FedAvg variants for quantization robustness and provide a convergence analysis for our Quantization-Aware variants in FL.
Overcoming Oscillations in Quantization-Aware Training
These effects are particularly pronounced in low-bit ($\leq$ 4-bits) quantization of efficient networks with depth-wise separable layers, such as MobileNets and EfficientNets.
Cyclical Pruning for Sparse Neural Networks
Current methods for pruning neural network weights iteratively apply magnitude-based pruning on the model weights and re-train the resulting model to recover lost accuracy.
Neural Network Quantization with AI Model Efficiency Toolkit (AIMET)
chapter 4) and quantization-aware training (QAT, cf.
Quantized sparse PCA for neural network weight compression
In this paper, we introduce a novel method of weight compression.
Understanding and Overcoming the Challenges of Efficient Transformer Quantization
Finally, we show that transformer weights and embeddings can be quantized to ultra-low bit-widths, leading to significant memory savings with a minimum accuracy loss.
A White Paper on Neural Network Quantization
Neural network quantization is one of the most effective ways of achieving these savings but the additional noise it induces can lead to accuracy degradation.
In-Hindsight Quantization Range Estimation for Quantized Training
Quantization techniques applied to the inference of deep neural networks have enabled fast and efficient execution on resource-constraint devices.
Bayesian Bits: Unifying Quantization and Pruning
We introduce Bayesian Bits, a practical method for joint mixed precision quantization and pruning through gradient based optimization.
Up or Down? Adaptive Rounding for Post-Training Quantization
In this paper, we propose AdaRound, a better weight-rounding mechanism for post-training quantization that adapts to the data and the task loss.
LSQ+: Improving low-bit quantization through learnable offsets and better initialization
To solve this problem, we propose LSQ+, a natural extension of LSQ, wherein we introduce a general asymmetric quantization scheme with trainable scale and offset parameters that can learn to accommodate the negative activations.
Ranked #18 on
Quantization
on ImageNet
Taxonomy and Evaluation of Structured Compression of Convolutional Neural Networks
The success of deep neural networks in many real-world applications is leading to new challenges in building more efficient architectures.
Data-Free Quantization Through Weight Equalization and Bias Correction
This improves quantization accuracy performance, and can be applied to many common computer vision architectures with a straight forward API call.
