Search Results for author:
Found 13 papers, 2 papers with code
ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals
By means of principal component analysis (PCA), it identifies a low-rank subspace (in practice 1/8 of the hidden dimension) in which activation variances are highest, and keep the coefficients within this subspace in high precision, e. g. 8-bit, while quantizing the rest to 4-bit.
Understanding the Difficulty of Low-Precision Post-Training Quantization for LLMs
Large language models of high parameter counts are computationally expensive, yet can be made much more efficient by compressing their weights to very low numerical precision.
Scaling Laws for Post Training Quantized Large Language Models
Generalization abilities of well-trained large language models (LLMs) are known to scale predictably as a function of model size.
Post Training Quantization of Large Language Models with Microscaling Formats
Large Language Models (LLMs) have distinguished themselves with outstanding performance in complex language modeling tasks, yet they come with significant computational and storage challenges.
Self-Selected Attention Span for Accelerating Large Language Model Inference
We demonstrate that using this custom CUDA kernel improves the throughput of LLM inference by 28%.
Mixed-Precision Quantization for Deep Vision Models with Integer Quadratic Programming
Layerwise bit-widths are assigned by optimizing a new MPQ formulation based on cross-layer quantization errors using an Integer Quadratic Program.
Laconic Deep Learning Computing
A Laconic configuration that uses a 1K-wire weight memory interface, outperforms the 2K-wire conventional accelerator by 15. 4x and is 1. 95x more energy efficient.
DPRed: Making Typical Activation and Weight Values Matter In Deep Learning Computing
The per group precisions are selected statically for the weights and dynamically by hardware for the activations.
Bit-Tactical: Exploiting Ineffectual Computations in Convolutional Neural Networks: Which, Why, and How
We show that, during inference with Convolutional Neural Networks (CNNs), more than 2x to $8x ineffectual work can be exposed if instead of targeting those weights and activations that are zero, we target different combinations of value stream properties.
Tartan: Accelerating Fully-Connected and Convolutional Layers in Deep Learning Networks by Exploiting Numerical Precision Variability
Experiments on image classification CNNs show that on average across all networks studied, TRT outperforms a state-of-the-art bit-parallel accelerator by 1:90x without any loss in accuracy while it is 1:17x more energy efficient.
Loom: Exploiting Weight and Activation Precisions to Accelerate Convolutional Neural Networks
LM can trade-off accuracy for additional improvements in execution performance and energy efficiency and compares favorably to an accelerator that targeted only activation precisions.
Dynamic Stripes: Exploiting the Dynamic Precision Requirements of Activation Values in Neural Networks
Stripes is a Deep Neural Network (DNN) accelerator that uses bit-serial computation to offer performance that is proportional to the fixed-point precision of the activation values.
Cnvlutin2: Ineffectual-Activation-and-Weight-Free Deep Neural Network Computing
We also present a modified organization that detects the activations that are deemed as ineffectual while fetching them from memory.
