改进4比特量化中注意力头的旋转策略,提升长文本生成效果
When Local Variance Optimality Is Not Enough: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisation
- 设计基于位置编码分解的局部旋转机制,避免全头混合带来的误差
- 实验表明该方法在短长上下文上均降低困惑度,优于传统哈达玛变换
- 适合关注高效量化与长序列生成质量的研究者
基于旋转的后训练量化通常对整个注意力头应用正交变换以减少异常值引起的误差。而位置编码(RoPE)将每个头划分为二维频率对,引发疑问:是否应采用尊重此分解的变换来优化性能?已有研究确立了与RoPE可交换的每对旋转。本文反向证明:对于不同频率,不存在其他单头正交映射能与RoPE可交换。针对实验中使用的头共享参数化,我们推导出在池化协方差、位置平均的代理目标下最小化较大通道方差的旋转角度,并验证实现达到理论最小值。在动态W4A4KV4设置下,所评估的头共享成对配置未提升精度;四个检查点中,用该配置替代全头哈达玛变换,在短和长上下文均增加困惑度。将成对旋转与哈达玛组合后,满足默认估计器下的±0.05困惑度区间标准。仅从键(K)估计共享角度可改善成对表现,但无法弥补与全头混合的差距。分析显示,解析目标控制的是池化校准协方差的位置平均二阶矩,而动态量化器则依据令牌级组范围设定步长。成对变换仅支持两通道混合。沿从两通道到全头混合的可控插值,随着混合支持增强,键值范围、相对量化误差及困惑度下降。结果表明,结构化代理的最优性未必降低量化误差,当代理与混合支持与量化器的尺度设定统计量不匹配时。
原文摘要 · Abstract (English)
Rotation-based post-training quantisation commonly applies an orthogonal transform across an entire attention head to reduce outlier-induced error. RoPE instead partitions each head into two-dimensional frequency pairs, raising the question of whether a transform respecting this decomposition can improve on full-head mixing. Prior work has established the per-pair rotations that commute with RoPE. We state the converse result that, for distinct frequencies, no other single-head orthogonal map commutes with RoPE. For the head-shared parameterisation used in our experiments, we then derive the rotation angle that minimises the larger channel variance under a pooled-covariance, position-averaged surrogate and verify that the implementation attains its analytic minimum. The evaluated head-shared pairwise configuration does not improve accuracy in the tested dynamic W4A4KV4 setting. Across four checkpoints, replacing the full-head Hadamard with this configuration increases perplexity at both short and long context lengths. Composing the pairwise rotation with the Hadamard satisfies the selected $\pm0.05$-PPL interval criterion under the default estimator. Estimating the shared angle from K alone improves pairwise-only on every checkpoint but does not close its gap to full-head mixing. The analytic objective controls a position-averaged second moment of a pooled calibration covariance, whereas the dynamic quantiser sets its step from a tokenwise group range. The pairwise transform also has only two-channel mixing support. Along a controlled interpolation from two-channel to full-head mixing, K range, relative quantisation error, and perplexity degradation decrease as support increases. These results show that optimality for a structured surrogate need not reduce quantisation error when the surrogate and mixing support are misaligned with the quantiser's scale-setting statistic.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。