提出诊断方法，精准定位长文本推理中缓存压缩失效原因
When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compression
- 用固定契约诊断法逐个测试缓存淘汰决策的可靠性
- 在三个模型上72.6%的有用缓存被正确识别，32.4%的无效缓存被误判
- 适合研究大模型长文本推理与缓存优化的工程师和研究员
长上下文大模型推理受解码时读取大型键值（KV）缓存带来的内存和带宽开销制约。KV压缩通过仅保留部分缓存来降低开销，但仅凭任务准确率无法判断选择器成功或失败的原因。选择器可能在三个环节出错：遗漏未来解码所需的证据、给不影响输出的标记高分、在将分数压缩到小缓存时破坏相关证据。本文提出一种固定契约诊断方法，保持选择器设置不变，逐个改变一个决策槽。针对价值排序，探针结合块的注意力质量与移除后输出变化的估计。在LongBench上，对三个模型和两种预算配置，探针在72.6%的正边际单元中为正，在32.4%的非正边际单元中为正。NeedleBench M-RT 32k和RULER 8k检查探针支持分支检索下的闭包性，264个单元的符号评估将支持恢复与输出价值排序从边界杠杆效应中分离。最终顺序为：恢复解码侧证据，评估其输出价值，并在投影中保留耦合证据。
原文摘要 · Abstract (English)
Long-context LLM inference is bottlenecked by the memory and bandwidth cost of reading large KV caches during decoding. KV compression reduces this cost by keeping only part of the cache, but task accuracy alone does not identify why a selector succeeds or fails. A selector can fail at three steps: it may miss the evidence future decoding needs, give high scores to tokens that do not affect the output, or break related evidence when fitting scores into a small cache. We introduce a fixed-contract diagnostic that holds the selector's setup fixed and changes one decision slot at a time. For value ranking, the probe combines a block's attention mass with the estimated output change from removing it. On LongBench across three models and two budgets, the probe is positive on 72.6% of positive-margin cells and 32.4% of nonpositive-margin cells. NeedleBench M-RT at 32k and a RULER 8k check probe support closure under branched retrieval, and a 264-cell sign evaluation separates support recovery and output-value ranking from leverage effects near the boundary. The resulting order is to recover decode-side evidence, rank its output value, and preserve coupled evidence during projection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。