提出无损量化新范式,实现模型压缩与推理加速兼得
Statistically-Lossless Quantization of Large Language Models

- 采用分层非均匀量化与非对称编码,突破传统量化瓶颈
- 3.3比特实现任务无损,5-6比特达成分布无损,速度提升1.7-3.7倍
- 提出可解释的精度指标EAR,适配高效部署与模型验证场景
模型量化已成为大语言模型高效部署的关键,但现有方法存在明显权衡:如GPTQ和AWQ等方法虽实现实用压缩却为有损,而无损技术虽保持精度却无法加速推理。本文探索统计无损压缩的中间路径,从三个层面定义“无损”:第一,任务无损压缩在激进位宽下仍保持零样本基准准确率(在自然采样方差内),第二,形式化更严格的分布无损压缩,要求量化模型的下一个词分布与原模型在统计上不可区分,并提出期望接受率(EAR)作为直接可解释的保真度指标,例如EAR≥0.99表示99%的词元一致率;第三,证明对称量化会引入γ²倍的噪声方差,因此分布无损必须使用非对称量化,而任务无损则无需此限制。通过SLQ方法,即基于分层非均匀量化、非对称编码与宽位搜索,在低于4比特/参数时实现任务无损,平均5-6比特/参数达到分布无损,且使用优化内核相比FP16实现1.7-3.7倍推理加速。代码已开源。
原文摘要 · Abstract (English)
Model quantization has become essential for efficient large language model deployment, yet existing approaches present clear trade-offs: methods such as GPTQ and AWQ achieve practical compression but are lossy, while lossless techniques preserve fidelity but lack inference acceleration. This paper explores the middle ground of statistically-lossless compression, examining three complementary aspects of what losslessness means for quantized LLMs. First, task-lossless compression preserves zero-shot benchmark accuracy within natural sampling variance and is achievable at aggressive bitwidths. Second, we formalize the stricter notion of distribution-lossless compression, requiring the quantized model's next-token distribution to be practically indistinguishable from the original, and propose the Expected Acceptance Rate (EAR), the maximum token-agreement probability under optimal coupling, as a directly interpretable fidelity metric. For example, EAR >= 0.99 means 99% agreement. Third, we prove a gamma-squared variance law showing that symmetric quantization inflates noise variance by gamma^2 relative to asymmetric quantization, making asymmetric quantization a prerequisite for distribution-lossless fidelity but not for task-level preservation. Through SLQ, a layer-wise non-uniform method with asymmetric quantization and wide bitwidth search, we obtain task-lossless compression at well below 4 bits per parameter, as low as 3.3 bits depending on the model, distribution-lossless compression at 5-6 bits per parameter on average, and inference speedups of 1.7-3.7x compared to FP16 using optimized kernels. Source code is available at [https://github.com/IST-DASLab/SLQ](https://github.com/IST-DASLab/SLQ).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。