通过哈达玛旋转与激活感知缩放，实现大模型权重的高精度联合二维量化。
QAM-W: Joint 2D Codebook Quantization for LLM Weights via Hadamard Rotation and Activation-Aware Scaling
- 将权重行进行归一化与哈达玛旋转，配对为二维坐标后统一量化。
- 在约5.5比特/权时，比BF16仅低0.4%困惑度，节省32%权重比特。
- 适合追求高精度压缩的大模型部署，尤其在5-6比特区间表现突出。
标量后训练量化器会丢失权重行内坐标的成对结构。我们提出QAM-W（权重的正交振幅调制），一种恢复该结构的编解码器：每行先做L2归一化，再进行分块哈达玛旋转，配对为二维坐标，使用基于单位圆高斯分布训练的单一Lloyd-Max码本进行量化，并引入激活感知的通道级缩放。在涵盖五个来自四个家族的LLM（参数量1.1B–13B）和八个量化配置的跨模型研究中，激活感知变体在约5.5比特/权时，所有模型在WikiText-2上的困惑度与BF16相差不超过±0.4%，达到SmoothQuant W8A8的质量水平，但权重比特减少32%。联合二维编码在同等比特率下比极坐标编码提升2–15个百分点的ΔPPL；配对KL与ΔPPL%在37个（方法, 模型）行上呈强相关（Spearman ρ=0.99），符合从编码失真到KL散度的单调复合界。3.5比特/权变体在抗量化架构上具竞争力；在严格4比特/权时，旋转码本前沿方法QTIP更优，但本文贡献在于5–6比特/权区间内的质量保障。
原文摘要 · Abstract (English)
Scalar post-training quantizers discard pairwise coordinate structure within weight rows. We introduce QAM-W (Quadrature Amplitude Modulation for Weights), a codec that recovers this structure: each row is L2-normalized, block-Hadamard rotated, paired into 2D coordinates, and quantized against a single Lloyd-Max codebook trained on the unit circular Gaussian, with activation-aware per-channel scaling. In a cross-model study spanning five LLMs from four families (1.1B--13B parameters) and eight quantized configurations, the activation-aware variant at $\approx 5.5$ bpw stays within $\pm 0.4\%$ of BF16 WikiText-2 perplexity on every model, matching the SmoothQuant W8A8 quality envelope at $32\%$ fewer weight bits. Joint 2D coding outperforms polar (amplitude $\times$ phase) coding by 2--15~pp $Δ$PPL at equal bitrate, and paired KL against BF16 tracks $Δ$PPL\% at Spearman $ρ= 0.99$ across 37 (method, model) rows, consistent with a monotone composite bound from codec distortion to KL divergence. A 3.5~bpw variant is competitive on quantization-tolerant architectures. At strict 4~bpw, the rotated-codebook frontier method QTIP outperforms QAM-W; the contribution is the quality-preserving 5--6~bpw band.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。