arXiv:2608.27128cs.CL2026-08

通过检测键值对重复性,智能修复长文本推理中的缓存淘汰漏洞。

TwinKV: A Composable Repair Pass for KV Cache Eviction via Pairwise Key Redundancy

  • 利用键值对近似重复性作为无训练、无需注意力的淘汰信号
  • 在压缩比0.3~0.7下提升多数策略性能,尤其在低冗余场景有效
  • 适合资源受限的小模型长文本推理,尤其对自回归生成有帮助

长上下文推理受键值(KV)缓存内存占用限制,尤其在小模型和严苛资源条件下。现有淘汰方法基于注意力分布或全局参考点距离打分,但通过留一法探测发现,注意力强度与令牌对答案的因果贡献无关(斯皮尔曼ρ = -0.004),挑战了主流方法的前提。本文提出TwinKV,一种无需训练、无需注意力的冗余信号,可检测某令牌键是否在上下文中存在近似重复。它不替代原有策略,而作为可组合的修复步骤:在固定保留集基础上,识别被丢弃但无副本的“孤儿”与信息被复制的“冗余捐赠者”,进行交换,保持原始预算与评分规则。在LongBench、LooGLE、RULER及MMLU-Pro上,针对Qwen3-4B和Llama-3.2-1B,在压缩比0.3、0.5、0.7下测试四类新淘汰策略。结果表明,对多数非饱和策略,TwinKV带来提升;在接近性能天花板的策略中效果有限;但在低压缩比时增益最小。值得注意的是,对于在所有评估单元均获提升的第四种策略(其Alone得分仍有改进空间),表现持续正向。整体上,Llama-3.2-1B在长文本任务平均增益较小,但更多任务单元改善,且在RULER上显著胜出。此外,少量示例分类任务中TwinKV无效。

原文摘要 · Abstract (English)

Long-context inference is bottlenecked by the memory footprint of the key-value (KV) cache, especially for small models under tight resource budgets. Existing KV cache eviction methods score tokens using the model's attention distribution or, in attention-free variants, each key's distance from a global reference point. Using a controlled leave-one-out probe, we find that attention magnitude is unrelated to a token's causal contribution to the answer (Spearman $ρ=-0.004$), challenging the premise behind dominant eviction methods. We introduce TwinKV, a training-free, attention-free redundancy signal that detects whether a token's key has a near-duplicate elsewhere in context. Rather than replacing existing policies, TwinKV acts as a composable repair pass: given a policy's fixed retained set, it identifies evicted tokens with no surviving duplicate (\emph{orphans}) and retained tokens whose information is duplicated elsewhere (\emph{redundant donors}), then swaps them while preserving the original budget and scoring rule. We compose TwinKV with four recent eviction policies across LongBench, LooGLE, RULER, and a short-context MMLU-Pro no-harm control at compression ratios ${0.3,0.5,0.7}$. On Qwen3-4B, TwinKV improves a majority of configurations for two policies, is near-even for a third, and helps only a minority for a fourth adaptive baseline already near a performance ceiling; gains across the three non-ceiling policies are smallest at the loosest ratio. On RULER with Llama-3.2-1B, however, that fourth policy improves in every evaluated cell because its Alone score leaves substantial room to improve. More broadly, Llama-3.2-1B shows a smaller average LongBench gain but a higher fraction of improved cells on LongBench and LooGLE than Qwen3-4B, plus a clean RULER win. We also identify few-shot classification exemplars as a task structure where TwinKV does not help on either model.

KV缓存长文本推理淘汰优化轻量化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。