通过观察注意力模式，延迟淘汰重要重复令牌，大幅降低长推理内存占用。
LazyEviction: Lagged KV Eviction with Attention Pattern Observation for Efficient Long Reasoning
- 基于注意力模式观察，延迟淘汰高频重现的关键词
- KV缓存减少50%~70%，精度几乎不变
- 适合长链条推理任务的高效内存优化
大语言模型通过思维链推理展现更强能力，但长推理序列带来显著的GPU内存开销，主要源于键值（KV）缓存增长。现有KV缓存压缩方法在长推理任务中表现不佳。本文分析推理过程中的注意力模式，发现存在‘令牌重要性周期性重现’现象：大量令牌在多次解码后重新获得高关注度，而现有方法未能捕捉此特性，可能导致关键令牌被意外淘汰。为此，我们提出LazyEviction，一种基于观察窗口的滞后淘汰框架，通过优先保留具有周期性出现特征的令牌，实现更智能的缓存管理。大量实验表明，LazyEviction可将KV缓存减少50%~70%，同时保持与基线相当的推理精度，优于现有压缩方法。代码已开源：https://github.com/Halo-949/LazyEviction。
原文摘要 · Abstract (English)
Large Language Models (LLMs) exhibit enhanced capabilities by Chain-of-Thought reasoning. However, the extended reasoning sequences introduce significant GPU memory overhead due to increased key-value (KV) cache. Existing KV cache compression methods mitigate memory bottlenecks but struggle in long reasoning tasks. In this paper, we analyze attention patterns in reasoning tasks and reveal a Token Importance Recurrence phenomenon: a large proportion of tokens regain high attention after multiple decoding steps, which is failed to capture by existing works and may lead to unpredictable eviction on such periodically critical tokens. To address this, we propose LazyEviction, an observation window-based lagged eviction framework retaining latent recurring tokens by prioritized eviction based on tokens' recurrence patterns. Extensive experiments demonstrate that LazyEviction reduces KV cache by 50%~70% while maintaining comparable accuracy, outperforming existing KV cache compression baselines. Our implementation code can be found at https://github.com/Halo-949/LazyEviction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。