arXiv:2608.25230cs.LGcs.CL2026-08

让被丢弃的键值对‘贡献’注意力权重,提升缓存效率

Trust the Mass: Forced Weights in KV-Cache Eviction

论文配图:Trust the Mass: Forced Weights in KV-Cache Eviction
图 1 · 摘自论文原文
  • 用被丢弃的键值对统计量生成无需训练的缓存分配策略
  • 在160组对比中胜过93组主流方法,仅在22组落后于基准
  • 适用于需高精度且内存受限的推理系统部署

每个部署的稀疏注意力或键值缓存淘汰规则都会保留部分键,丢弃其余,并对保留集重新归一化注意力权重。在五个模型的168,192个注意力行上枚举最优子集发现,仅保留最大权重已接近最优,最佳选择仅缩小了中位数2%至5%的与全注意力的差距。若选择策略提升有限,则现有方法的性能差异应源于其他因素,因此我们测量各方法所占用字节数。在统一评估流程中,最强的无查询依赖方法因每头选择以掩码形式存储而占据完整缓存,唯有采用不规则每头存储才能释放内存。对单一固定选择施加名义预算将导致14至62个基准点损失。我们发现87.6分的检索优势源自问题可见时的排名计算。ContourKV是一种无需训练的分配器,基于被丢弃质量统计构建,在160组配对比较中胜出93次,仅落后22次于预算约束基线,且与最强基线持平。

原文摘要 · Abstract (English)

Every deployed sparse-attention or KV-cache-eviction rule keeps a subset of the keys, discards the rest, and renormalizes the attention weights over the kept set. Enumerating the exact best subset under that constraint on $168{,}192$ attention rows from five models shows that keeping the largest weights is already near-optimal, since the best subset closes only a median $2$ to $5\%$ of the remaining gap to full attention. If selection closes this little, published margins between eviction methods must come from elsewhere, so we measure the bytes each method holds. In the shared evaluation pipeline, the strongest query-agnostic methods hold the full cache because their per-head selections are stored as masks, and only ragged per-head storage frees that memory. Enforcing a nominal budget on one fixed selection costs $14$ to $62$ benchmark points. We trace an $87.6$-point retrieval margin to rankings computed while the question is visible. ContourKV, a training-free allocator built from the dropped-mass statistic, wins $93$ of $160$ paired comparisons against that state of the art and loses $22$ at the byte count of the budget-enforcing baselines, and it ties the strongest of them.

KV缓存注意力机制推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。