arXiv:2608.05326cs.LGcs.CL2026-08被引 1

提出可恢复的量化缓存淘汰机制,提升长文本生成时的上下文保留能力。

QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding

论文配图:QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding
图 1 · 摘自论文原文
  • 引入可恢复淘汰策略,将缓存分为三档:高置信度全精度、中等置信度量化存储、低置信度删除。
  • 在固定内存下,相比基线方法减少30%以上被遗漏的重要注意力,信息保留率显著提升。
  • 适合需要长期依赖建模的场景,如长文档理解、复杂推理任务。

自回归大模型推理受键值(KV)缓存内存占用限制日益严重。主流方法通过注意力分数判断并淘汰不重要词元以减小缓存。然而这类策略隐含不可逆假设:一旦淘汰,无法重新启用。我们发现此假设在解码过程中脆弱——随着生成查询演化,词元与窗口的重要性会漂移,导致标准淘汰策略永久丢弃后续可能获得强注意力的缓存状态。为此,我们引入未来遗漏质量(Future Missed Mass)和全局重激活率(Global LIR)两个诊断指标,衡量被丢弃状态未来的注意力分配及历史冷区的重新激活。提出QEvict,一种三层次KV缓存管理方案:高置信度窗口保持全精度,中等置信度窗口以量化形式可恢复存储,低置信度窗口直接删除。解码期间,累积注意力分数动态更新重要性;当量化窗口再次变得重要时,将其反量化并提升至全精度层。在固定内存预算下,该设计在保留更广历史上下文的同时,对最关键区域维持精确全精度。在长上下文理解、检索与推理基准测试中,QEvict持续优于代表性淘汰与量化基线,显著降低遗漏注意力,提升信息保留。

原文摘要 · Abstract (English)

Autoregressive large language model inference is increasingly constrained by the memory footprint of the Key-Value (KV) cache. A dominant line of work reduces this footprint by evicting tokens that appear unimportant under attention-derived scores. However, such policies make an implicit irreversible decision: once a token is evicted, it cannot become useful again. We show that this assumption is brittle during decoding. Token and window importance drift as generated queries evolve, causing standard eviction policies to permanently discard states that later receive substantial attention under the full-cache model. To characterize this behaviour, we introduce Future Missed Mass and Global LIR, two diagnostics that measure future attention assigned to discarded states and the reactivation of historically inactive regions. We propose QEvict, a three-tier KV-cache management scheme that replaces binary retain-or-delete eviction with recoverable eviction. QEvict maintains high-confidence windows in full precision, stores intermediate windows in a quantized recoverable tier, and deletes only the lowest-confidence windows. During decoding, cumulative attention scores update window importance and when a quantized window becomes important again, it is dequantized and promoted to the full-precision. Under a fixed memory budget, this design preserves broader historical context while retaining exact full precision for the most important regions. Across long-context understanding, retrieval, and reasoning benchmarks, QEvict consistently improves over representative eviction and quantization baselines, reducing missed attention and improving information retention

KV缓存长文本生成量化可恢复淘汰

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。