用未来注意力信号训练缓存淘汰策略，大幅压缩内存占用同时保持推理质量。
KVpop -- Key-Value Cache Compression with Predictive Online Pruning
- 通过未来注意力目标直接监督缓存保留或丢弃决策
- 在Qwen3-4B上实现75%压缩率下保留98%性能，88%压缩率达97%
- 首次提出延迟评分机制，利用近期上下文提升淘汰准确性
键值（KV）缓存增长是自回归解码的主要瓶颈，因内存与带宽随上下文长度线性增加。现有缓存淘汰方法多依赖静态启发式或代理评分，难以准确追踪未来标记的有用性，导致相关性变化时淘汰策略脆弱。为此，我们提出KVpop，通过直接监督保留/丢弃决策来学习固定预算的缓存淘汰策略。该评分器基于一种新型未来注意力目标进行训练，无需生成密集注意力图即可高效计算。此外，我们引入一种延迟记忆评分器，是首个在学习型淘汰方法中延迟评分固定步数以利用近未来上下文的方法。在AIME和HMMT数学推理任务上，KVpop在Qwen3-4B上实现75%缓存压缩率时保留98%全注意力性能，88%压缩率下达97%，持续优于现有基线。Qwen3-8B表现更优，接近全教师模型性能。结果表明，以未来注意力信号监督淘汰可显著降低内存开销，同时维持高质量推理。
原文摘要 · Abstract (English)
Key-value (KV) cache growth is a major bottleneck in autoregressive decoding, as memory and bandwidth scale linearly with context length. Existing KV eviction methods often rely on static heuristics or proxy scores, which poorly track future token utility and cause brittle eviction as relevance shifts. To address this, we introduce KVpop, which learns a fixed-budget KV eviction policy by directly supervising the keep-or-drop decision. The scorer is trained against a novel future-attention target, computed efficiently without materializing dense attention maps. We further introduce a delayed memory-based scorer that, uniquely among learned eviction methods, defers scoring for a fixed number of steps to exploit near-future context. On AIME and HMMT mathematical reasoning, KVpop retains 98% of full-attention performance on Qwen3-4B at 75% KV cache compression and 97% at 88% compression, consistently outperforming established eviction baselines. Qwen3-8B shows even stronger results, reaching near-full teacher performance. These results show that supervising eviction with future-attention signals cuts memory costs while maintaining quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。