量化比降维更有效压缩注意力缓存，精度损失更小。
Quantization Dominates Rank Reduction for KV-Cache Compression
- 用量化替代降维来压缩注意力缓存，保留所有维度但降低精度。
- 在相同存储下，量化比降维平均高出4-364的PPL，GQA越激进差距越大。
- 适合追求高精度低延迟推理的模型部署场景，尤其大模型应用。
我们比较了Transformer推理中压缩键值缓存的两种策略：降维（丢弃维度）和量化（保留所有维度但降低精度）。在五个模型（124M-14B，MHA与GQA）上，相同存储预算下，量化始终优于降维，性能差距达4-364 PPL，且即使结合使用，差距仍存在并随GQA强度增加而扩大。在LAMBADA测试中，INT4量化可达到FP16精度（Mistral 7B仅+0.23 PPL，GPT-2为+0.58），而同等存储下秩为32的降维性能暴跌至0.4%。原因在于结构不对称：降维可能使注意力分配发生离散突变，而量化噪声有界且通常保持分数排序。通过扰动分析发现，在softmax Fisher度量下，投影损伤比量化损伤高3×2^(2b)倍/方向。基底消融实验确认该优势与坐标系无关（差异<0.4 PPL），表明关键在于保留维度而非选择更好基。联合对K+V进行INT4量化可在Mistral 7B上实现75%总缓存压缩，仅带来+0.18 PPL增益。
原文摘要 · Abstract (English)
We compare two strategies for compressing the KV cache in transformer inference: rank reduction (discard dimensions) and quantization (keep all dimensions, reduce precision). At matched storage budgets across five models (124M-14B, MHA and GQA), we find that quantization consistently outperforms rank reduction by 4-364 PPL depending on model and compression level. The gap persists even when rank reduction is combined with quantization in hybrid baselines, and it grows with GQA aggressiveness. On LAMBADA, INT4 matches FP16 accuracy (+0.23 PPL on Mistral 7B, +0.58 on GPT-2) while rank-32 at identical storage collapses to 0.4%. We trace this gap to a structural asymmetry: under softmax attention routing, removing a dimension can flip which token is attended (a discrete failure), while quantization noise is bounded and typically preserves score ordering. We formalize this via a perturbation result showing projection damage exceeds quantization damage by 3 x 2^(2b) per direction under the softmax Fisher metric. A basis ablation confirms the finding is basis-independent (spread <0.4 PPL), establishing that the advantage comes from preserving dimensions, not from a better coordinate system. Joint K+V INT4 quantization achieves 75% total KV reduction at only +0.18 PPL on Mistral 7B.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。