Hyper-Compression: Model Compression via Hyperfunction
The rapid growth of large models' size has far outpaced that of GPU memory. To bridge this gap, inspired by the parsimonious relationship between genotype and phenotype, we turn the model compression problem into the issue of parameter representation to propose the so-called hyper-compression. The hyper-compression uses a hyperfunction to represent the parameters of the target network per ergodic theory, that addresses the following approximation problem: if a low-dimensional dynamic system can fill the high-dimensional space eventually. Empirically, the proposed hyper-compression enjoys the following merits: 1) \textbf{P}referable compression ratio; 2) \textbf{N}o post-hoc retraining; 3) \textbf{A}ffordable inference time; and 4) \textbf{S}hort compression time. It compresses LLaMA2-7B in an hour and achieves close-to-int4-quantization performance, without retraining and with a performance drop of less than 1\%. Our work can facilitate the harmony between the scaling law and the stagnation of hardware upgradation in terms of saving both computation and data. We have open-sourced our \href{https://github.com/Juntongkuki/Hyper-Compression.git}{code} for readers' free download and evaluation.
PDF Abstract
CIFAR-10
MMLU
Natural Questions
HellaSwag
BoolQ
WinoGrande
SciQ
