提出新量化方法RDQ，解决大模型4比特以下精度下降问题。
RDQ: Residual Distribution Quantization for Large Language Models
- 通过级联误差补偿校准每层残差分布漂移，提升量化精度。
- 在LLaMA-3-8B上实现7.55/5.62困惑度（W3/W4），优于现有最佳方法。
- 无需运行时开销，兼容主流推理框架，适合部署场景。
大语言模型后训练量化在4比特以下精度时性能急剧下降。我们发现根本原因是残差流分布漂移：每层变换引入的量化噪声在共享残差表示中累积，导致与FP16基线的KL散度随深度超线性增长（与对数困惑度相关系数r=0.999，p<0.001），且在所有测试方法和比特位宽下均成立。我们发现84%的LLaMA-3-8B层呈现非高斯残差分布（KS检验，p≤0.05），且每层残差方差随深度增长6,548倍。提出RDQ（残差分布量化）框架，核心是级联误差补偿（CEC）：一种顺序校准过程，通过运行校准数据通过已量化上游层来捕获实际接收到的漂移激活，并针对这些漂移输入拟合通道级AWQ风格缩放系数，再将缩放系数折叠至前序RMSNorm权重中，实现零推理开销下的数学等价。RDQ在三个测试架构上均达最优：LLaMA-3-8B（W3/W4：7.55/5.62 PPL）、Qwen-2.5-7B（7.46/6.38 PPL）、Mistral-7B（6.88/5.73 PPL）。在所有模型和比特组合下均超越现有最优基线（LeanQuant/SpinQuant），在LLaMA-3-8B W3A16上相比RTN提升高达-46.4%。所有输出均为标准组大小128的非对称量化，可部署于Qualcomm AIMET、GGUF及任何标准推理栈，零运行时开销。
原文摘要 · Abstract (English)
Post-training quantization (PTQ) of large language models degrades sharply below 4-bit precision. We identify the root cause as residual stream distributional drift: quantization noise injected at each transformer layer accumulates in the shared residual representation, causing KL divergence from the FP16 baseline to grow super-linearly with depth (Pearson r=0.999 with log-perplexity, p<0.001, confirmed across all tested methods and bit-widths). We discover that 84% of LLaMA-3-8B layers exhibit non-Gaussian residual distributions (KS test, p<=0.05), and that per-layer residual stream variance grows 6,548x across depth. We propose RDQ (Residual Distribution Quantization), a PTQ framework whose central contribution is Cascaded Error Compensation (CEC): a sequential calibration procedure that captures the actual drifted activations each layer receives (computed by running calibration data through already-quantized upstream layers) and fits per-channel AWQ-style scales against those drifted inputs, with scales folded into preceding RMSNorm weights for exact mathematical equivalence at zero inference overhead. RDQ achieves state-of-the-art results on all three tested architectures: LLaMA-3-8B: 7.55 / 5.62 PPL (W3/W4); Qwen-2.5-7B: 7.46 / 6.38 PPL; Mistral-7B: 6.88 / 5.73 PPL. RDQ beats the best published baseline (LeanQuant/SpinQuant) at every model and bit-width combination, with gains up to -46.4% vs. RTN at W3A16 on LLaMA-3-8B. All output is standard group-128 asymmetric quantization, deployable on Qualcomm AIMET, GGUF, and any standard inference stack at zero runtime overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。