用动态系统生成参数,实现无需重训的高效模型压缩。
Hyper-Compression: Model Compression via Hyperfunction

- 用无理绕转动态系统作为超函数,以轨迹长度表示模型参数。
- 压缩LLaMA2-7B仅需1小时,性能损失低于1%,接近int4量化效果。
- 适合追求快速部署、低延迟推理的工业级模型压缩场景。
大型模型规模的快速增长已远超计算资源的提升速度。受大脑发育中基因型与表型间简约关系的启发,我们提出一种名为超压缩(Hyper-Compression)的新机制,将模型压缩问题转化为通过超函数进行参数表示。具体而言,某些低维动态系统的轨迹最终可填满高维空间,因此利用此类系统作为超函数,以对应组合数或轨迹长度表示目标网络的参数。这提供了一种与剪枝、量化、蒸馏和分解均不同的全新压缩范式。我们方法性地识别出具有无理绕转特性的动态系统作为合适的超函数,并理论推导了其关联的误差界。基于理论洞察,设计多种工程优化使该方法实用有效。在包括LLaMA和Qwen系列在内的NLP模型及视觉模型上进行系统实验表明,该方法具备四大优势:(1) 可喜的压缩比;(2) 无需事后微调;(3) 推理开销可控;(4) 压缩时间短。例如,可在一小时内压缩LLaMA2-7B,性能损失小于1%,接近int4量化表现。代码已开源:https://github.com/Juntongkuki/Hyper-Compression.git。
原文摘要 · Abstract (English)
The rapid growth of large models' size has far outpaced that of computing resources. To bridge this gap, encouraged by the parsimonious relationship between genotype and phenotype in the brain's growth and development, we propose the so-called Hyper-Compression that turns the model compression into the issue of parameter representation via a hyperfunction. Specifically, it is known that the trajectory of some low-dimensional dynamic systems can fill the high-dimensional space eventually. Thus, Hyper-Compression, using these dynamic systems as the hyperfunctions, represents the parameters of the target network by their corresponding composition number or trajectory length. This suggests a novel mechanism for model compression, substantially different from the existing pruning, quantization, distillation, and decomposition. Along this direction, we methodologically identify a suitable dynamic system with the irrational winding as the hyperfunction and theoretically derive its associated error bound. Next, guided by our theoretical insights, we propose several engineering twists to make the Hyper-Compression pragmatic and effective. Lastly, systematic and comprehensive experiments on \textcolor{black}{NLP models such as LLaMA and Qwen series and vision models} confirm that Hyper-Compression enjoys the following \textbf{PNAS} merits: 1) \textbf{P}referable compression ratio; 2) \textbf{N}o post-hoc retraining; 3) \textbf{A}ffordable inference time; and 4) \textbf{S}hort compression time. It compresses LLaMA2-7B in an hour and achieves close-to-int4-quantization performance, without retraining and with a performance drop of less than 1\%. We have open-sourced our code in https://github.com/Juntongkuki/Hyper-Compression.git for free download and evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。