arXiv:2607.29591cs.CL2026-07

提出ResKV,用残差缓存重建被丢弃的注意力贡献,提升长文本推理效率。

ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression

  • 将缓存分为精确主缓存和紧凑残差缓存,重建被剔除项的影响
  • 在相同保留缓存预算下,相比基线显著提升长文本推理性能
  • 适用于多种模型、缓存预算及压缩方法,适合实际部署

KV缓存压缩对高效长上下文推理至关重要。现有淘汰方法永久丢弃未选中的标记,从而移除其对注意力的累积贡献;基于融合的方法虽保留更多信息,但可能扰动应保持精确的键值。我们观察到,缓存淘汰所丢失的信息可形式化为softmax注意力中分子和分母的残差统计量。基于此,我们提出ResKV,将固定缓存预算划分为精确主缓存与紧凑残差缓存,后者重建被舍弃标记的贡献。主缓存项与残差条目共同参与同一softmax归一化,使残差项同时恢复注意力分子与分母的质量,而非作为事后修正。构建时的验证代理决定各层与注意力头的残差分配,解码时的动态门控则按查询调整残差贡献。在LongBench和RULER上的全面评估显示,在相同保留缓存预算下,无论查询感知还是查询无关设置,多种骨干网络、缓存预算及代表性压缩基线均取得广泛改进,同时保持压缩解码的实际效率,包括峰值内存占用与长上下文解码吞吐量。

原文摘要 · Abstract (English)

KV cache compression is essential for efficient long-context inference. Existing eviction methods permanently discard unselected tokens and consequently remove their aggregate contribution to attention. Merging-based alternatives preserve more information but can perturb retained keys and values that should remain exact. We observe that the information omitted by cache eviction can be formulated as residual statistics in both the numerator and denominator of softmax attention. Based on this observation, we propose ResKV, which divides a fixed KV budget into an exact main cache and a compact residual cache that reconstructs the contribution of omitted tokens. ResKV lets main-cache tokens and residual entries participate in the same softmax normalization, so residual entries restore both attention numerator and denominator mass rather than acting as a post-hoc correction. A construction-time validation proxy determines residual allocation for each layer and KV head, while a decode-time dynamic gate adjusts residual contributions for individual queries. Comprehensive evaluations on LongBench and RULER, covering query-aware and query-agnostic settings, multiple backbones, cache budgets, and representative compression baselines, demonstrate broad improvements under the same retained KV budget while preserving the practical efficiency of compressed decoding, including peak memory usage and long-context decode throughput.

KV缓存注意力机制长文本推理缓存压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。