提出ASER算法，提升大模型低比特量化精度。
ASER: Activation Smoothing and Error Reconstruction for Large Language Model Quantization
- 通过误差重建与激活平滑减少量化误差。
- 在W4A8每通道设置下保持高精度，优于现有方法。
- 适合追求高效部署的大模型应用者。
量化是大语言模型服务中的关键技术，但在实现低比特量化时面临严峻挑战。有限的数值映射导致量化模型产生显著误差，造成不可接受的性能下降。本文基于模型压缩目标，深入分析后训练量化过程中大模型各层的误差分布。提出ASER算法，包含：(1) 误差重建：通过白化SVD构造类似LoRA的低秩矩阵补偿量化误差；(2) 激活平滑：提取异常值以获得更平滑的激活，提升误差补偿效果。ASER可将典型大模型有效量化至低比特，尤其在W4A8每通道设置下仍能保持高精度。实验表明，ASER在主流量化算法中表现优异，具备向激活量化的拓展潜力，且开销极小。
原文摘要 · Abstract (English)
Quantization stands as a pivotal technique for large language model (LLM) serving, yet it poses significant challenges particularly in achieving effective low-bit quantization. The limited numerical mapping makes the quantized model produce a non-trivial error, bringing out intolerable performance degration. This paper is anchored in the basic idea of model compression objectives, and delves into the layer-wise error distribution of LLMs during post-training quantization. Subsequently, we introduce ASER, an algorithm consisting of (1) Error Reconstruction: low-rank compensation for quantization error with LoRA-style matrices constructed by whitening SVD; (2) Activation Smoothing: outlier extraction to gain smooth activation and better error compensation. ASER is capable of quantizing typical LLMs to low-bit ones, particularly preserving accuracy even in W4A8 per-channel setup. Experimental results show that ASER is competitive among the state-of-the-art quantization algorithms, showing potential to activation quantization, with minor overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。