arXiv:2510.05176cs.LGcs.AI2025-10

通过模式对齐量化,让长文本推理的键值缓存更省显存、更快还更准。

PatternKV: Flattening KV Representation Expands Quantization Headroom

  • 在线挖掘模式向量,只量化与模式的偏差,让分布更平坦。
  • 2比特下性能提升明显,4比特误差仅0.08%,测试时扩展准确率高10%。
  • 适合大上下文、高并发推理场景,特别适合资源受限部署。

自回归大模型中的键值缓存(KV cache)虽避免重复计算,却成为推理阶段内存和带宽的主要瓶颈,尤其在长上下文和测试时扩展场景下。现有量化方法聚焦于分离异常值,但无法整体摊平分布,导致低比特下性能脆弱。本文发现:键向量(K)具有稳定、随上下文演化的结构,值向量(V)承载潜在语义规律,二者共同形成共享模式。基于此,提出PatternKV:在线挖掘代表性模式向量,将每个KV向量对齐至最近模式,并仅量化残差部分。该方法重塑了KV分布,使其更平坦、范围更窄,显著提升低比特量化保真度。在多个骨干模型上,模式在长上下文与测试时扩展设置下均表现优异:2比特性能显著提升,4比特平均误差仅0.08%(相比FP16),测试时扩展准确率平均提升10%,吞吐提升1.5倍,支持1.25倍更大批量。

原文摘要 · Abstract (English)

KV cache in autoregressive LLMs eliminates redundant recomputation but has emerged as the dominant memory and bandwidth bottleneck during inference, notably with long contexts and test-time scaling. KV quantization is a key lever for reducing cache cost, but accuracy drops sharply as the native KV distribution lacks flatness and thus maintains a wide quantization range. Prior work focuses on isolating outliers, which caps their error but fails to flatten the overall distribution, leaving performance fragile under low-bit settings. In this work, we show that the K cache maintains a stable, context-evolving structure, while the V cache carries latent semantic regularities, with both contributing to the organization of vectors into shared patterns. Building on these insights, we propose PatternKV, a pattern-aligned residual quantization scheme. It mines representative pattern vectors online, aligns each KV vector to its nearest pattern, and quantizes only the residual. This reshaping of the KV distribution flattens the quantization target and narrows its range, thereby improving the fidelity of low-bit KV quantization. Across long-context and test-time scaling settings on multiple backbones, PatternKV delivers consistent 2-bit gains, with a 0.08% average 4-bit drop relative to FP16, improves test-time scaling accuracy by 10% on average, and raises throughput by 1.5x while supporting 1.25x larger batches.

KV量化长文本生成推理加速低比特

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。