提出新视角解析量化误差,提升大模型压缩精度。
Dissecting Quantization Error: A Concentration-Alignment Perspective
- 从信号噪声比拆解量化误差,关注权重与激活的分布集中度和方向对齐。
- 引入轻量级块变换CAT,在4比特下优于现有方法。
- 适合关注模型量化效率与精度平衡的研究者。
量化可显著提升大语言和视觉模型的效率,但通常导致精度下降。近期,保持函数不变的变换(如旋转、哈达玛变换、通道缩放)被成功用于降低训练后量化误差,但其原理尚不清晰。本文通过信号-量化噪声比(SQNR)分析线性层量化,发现固定比特宽度下,SQNR可分解为:(i) 权重与激活的集中度(反映分布范围与异常值),(ii) 其主变异性方向的对齐度。这揭示了一个可操作的洞见:除集中度外,提升权重与激活方向对齐可进一步减少量化误差。基于此,我们提出块集中-对齐变换(CAT),利用小校准集的协方差估计,联合优化集中度与对齐度,近似最大化SQNR。在多个LLM上的实验表明,CAT在4比特精度下始终匹配或超越已有变换量化方法,验证了框架的有效性。
原文摘要 · Abstract (English)
Quantization can drastically increase the efficiency of large language and vision models, but typically incurs an accuracy drop. Recently, function-preserving transforms (e.g. rotations, Hadamard transform, channel-wise scaling) have been successfully applied to reduce post-training quantization error, yet a principled explanation remains elusive. We analyze linear-layer quantization via the signal-to-quantization-noise ratio (SQNR), showing that for uniform integer quantization at a fixed bit width, SQNR decomposes into (i) the concentration of weights and activations (capturing spread and outliers), and (ii) the alignment of their dominant variation directions. This reveals an actionable insight: beyond concentration - the focus of most prior transforms (e.g. rotations or Hadamard) - improving alignment between weight and activation can further reduce quantization error. Motivated by this, we introduce block Concentration-Alignment Transforms (CAT), a lightweight linear transformation that uses a covariance estimate from a small calibration set to jointly improve concentration and alignment, approximately maximizing SQNR. Experiments across several LLMs show that CAT consistently matches or outperforms prior transform-based quantization methods at 4-bit precision, confirming the insights gained in our framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。