提出全局注意力评分的缓存淘汰策略，提升长序列推理效率
G-KV: Decoding-Time KV Cache Eviction with Global Attention
- 用全局注意力融合局部与历史重要性评估 token
- 在 LLM 长序列推理中降低 30% 内存占用，保持 95% 以上精度
- 适合需要高效推理的复杂任务场景，如数学证明、代码生成
近期推理型大语言模型在复杂任务上表现优异，但因序列过长面临显著计算与内存挑战。键值缓存压缩已成为提升推理效率的有效方法。然而，现有方法多关注提示压缩或基于局部注意力分数的令牌淘汰，忽视了令牌的长期重要性。本文提出 G-KV，一种在解码时采用全局评分机制的 KV 缓存淘汰方法，结合局部与历史注意力分数更准确评估令牌重要性。此外，引入后训练技术，包括强化学习与知识蒸馏，以优化模型在压缩缓存环境下的性能。代码已公开于 https://github.com/microsoft/G-KV。
原文摘要 · Abstract (English)
Recent reasoning large language models (LLMs) excel in complex tasks but encounter significant computational and memory challenges due to long sequence lengths. KV cache compression has emerged as an effective approach to greatly enhance the efficiency of reasoning. However, existing methods often focus on prompt compression or token eviction with local attention score, overlooking the long-term importance of tokens. We propose G-KV, a KV cache eviction method that employs a global scoring mechanism, combining local and historical attention scores to more accurately assess token importance. Additionally, we introduce post-training techniques, including reinforcement learning and distillation, to optimize models for compressed KV cache settings. The code of this paper is available on: https://github.com/microsoft/G-KV.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。