通过误差补偿联合优化量化与稀疏化，实现大模型高效压缩。
Optimal Brain Restoration for Joint Quantization and Sparsification of LLMs
- 利用二阶海森矩阵构建误差补偿机制，协调量化与稀疏的矛盾需求。
- 在现有大模型上实现W4A4KV4量化+50%稀疏，速度提升4.72倍，内存减少6.4倍。
- 训练无关、通用性强，适合部署阶段的模型压缩场景。
大型语言模型（LLM）压缩近年来取得显著进展，如量化和剪枝。然而，当这些技术逐渐逼近其极限时，仅依赖单一方法进一步压缩变得愈发困难。本文探索将量化与稀疏化联合使用的新路径。尽管该联合方法前景广阔，但因其对权重分布要求本质冲突——量化偏好紧凑范围，剪枝则受益于高方差——带来新挑战。为此，我们提出无需训练的通用框架Optimal Brain Restoration（OBR），通过在剪枝与量化间进行误差补偿来协调二者。OBR基于二阶海森目标函数，经代理近似后转化为可解问题，并通过分组误差补偿获得闭式解。实验表明，OBR可在现有大模型上实现W4A4KV4量化与50%稀疏，相较FP16稠密基线，最高实现4.72倍加速与6.4倍内存压缩。
原文摘要 · Abstract (English)
Recent advances in Large Language Model (LLM) compression, such as quantization and pruning, have achieved notable success. However, as these techniques gradually approach their respective limits, relying on a single method for further compression has become increasingly challenging. In this work, we explore an alternative solution by combining quantization and sparsity. This joint approach, though promising, introduces new difficulties due to the inherently conflicting requirements on weight distributions: quantization favors compact ranges, while pruning benefits from high variance. To attack this problem, we propose Optimal Brain Restoration (OBR), a general and training-free framework that aligns pruning and quantization by error compensation between both. OBR minimizes performance degradation on downstream tasks by building on a second-order Hessian objective, which is then reformulated into a tractable problem through surrogate approximation and ultimately reaches a closed-form solution via group error compensation. Experiments show that OBR enables aggressive W4A4KV4 quantization with 50% sparsity on existing LLMs, and delivers up to 4.72x speedup and 6.4x memory reduction compared to the FP16-dense baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。