提出新量化方法HeadQ，显著降低大模型推理困惑度。
HeadQ: Model-Visible Distortion and Score-Space Correction for KV-Cache Quantization
- 在得分空间修正键值缓存误差，而非仅优化存储重建
- 2比特量化下减少84%~94%的额外困惑度，6个模型均有效
- 适合追求高精度低比特量化的大模型部署场景
KV缓存量化通常优化存储空间重建，但注意力机制通过逻辑值读取键，通过加权读取值。我们主张应以模型可见坐标衡量持续缓存误差。对键而言，可见误差为得分误差模常数偏移；由此提出HeadQ，一种键侧方法，利用校准学习的查询基底存储低秩残差编码，并作为加性逻辑修正。对值而言，固定注意力读取提供$A^2$加权的词元失真代理。在六个模型中，费舍尔/得分空间误差比原始键均方误差更准确预测注意力KL；相同预算反例、零空间干预、查询主成分分析控制及错误符号的HeadQ均证伪了存储均方误差的替代方案。匹配的Pythia检查点将主要异常定位至小模型低熵路径翻转边界。在仅键的WikiText-103解码实验中，2比特密集值设置下，HeadQ消除强2比特行约84%~94%的额外困惑度；在辅助全键值2比特组合中，HeadQ结合$A^2$值策略提升全部六种模型性能。
原文摘要 · Abstract (English)
KV-cache quantizers usually optimize storage-space reconstruction, even though attention reads keys through logits and values through attention-weighted readout. We argue that persistent cache error should be measured in model-visible coordinates. For keys, the visible object is score error modulo constant shifts; this yields HeadQ, a key-side method that stores a low-rank residual side code in a calibration-learned query basis and applies it as an additive logit correction. For values, fixed-attention readout gives an $A^2$-weighted token-distortion surrogate. Across six models, Fisher/score-space error predicts attention KL far better than raw key MSE; same-budget counterexamples, null-space interventions, query-PCA controls, and wrong-sign HeadQ falsify storage-MSE alternatives. Matched Pythia checkpoints localize the main anomaly to a small-model low-entropy route-flip boundary. In K-only WikiText-103 decode experiments with dense values, HeadQ removes roughly $84$--$94\%$ of the excess perplexity on the strongest 2-bit rows; in an auxiliary full-KV 2-bit composition, HeadQ plus an $A^2$ value policy improves all six models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。