Search Results for author:
Found 8 papers, 1 papers with code
EoRA: Training-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation
In this work, we re-formulate the model compression problem into the customized compensation problem: Given a compressed model, we aim to introduce residual low-rank paths to compensate for compression errors under customized requirements from users (e. g., tasks, compression ratios), resulting in greater flexibility in adjusting overall capacity without being constrained by specific compression formats.
ESPACE: Dimensionality Reduction of Activations for Model Compression
The activation-centrality of the approach enables retraining LLMs with no loss of expressivity; while at inference, weight decomposition is obtained as a byproduct of matrix multiplication associativity.
Optimal Clipping and Magnitude-aware Differentiation for Improved Quantization-aware Training
Data clipping is crucial in reducing noise in quantization operations and improving the achievable accuracy of quantization-aware training (QAT).
HarDNN: Feature Map Vulnerability Evaluation in CNNs
As Convolutional Neural Networks (CNNs) are increasingly being employed in safety-critical applications, it is important that they behave reliably in the face of hardware errors.
Accumulation Bit-Width Scaling For Ultra-Low Precision Training Of Deep Networks
Observing that a bad choice for accumulation precision results in loss of information that manifests itself as a reduction in variance in an ensemble of partial sums, we derive a set of equations that relate this variance to the length of accumulation and the minimum number of bits needed for accumulation.
Per-Tensor Fixed-Point Quantization of the Back-Propagation Algorithm
The high computational and parameter complexity of neural networks makes their training very slow and difficult to deploy on energy and storage-constrained computing systems.
Analytical Guarantees on Numerical Precision of Deep Neural Networks
We focus on numerical precision – a key parameter defining the complexity of neural networks.
Understanding the Energy and Precision Requirements for Online Learning
Lower bounds on the data precision are derived in terms of the the desired classification accuracy and precision of the hyperparameters used in the classifier.
