arXiv:2603.10899cs.LGcs.AI2026-03被引 6

不生成草稿就能提前预判关键信息,高效清理无关缓存。

LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation

  • 用轻量模块预测未来重要性,无需昂贵的草稿生成。
  • 在长文本任务中提升准确率,缓存淘汰成本降低14.5倍。
  • 适合追求快速响应的实时大模型部署场景。

基于Transformer的大语言模型在自回归推理中依赖键值(KV)缓存以避免重复计算。然而,缓存大小随输入序列长度线性增长,成为长上下文任务的瓶颈。现有方法通过剔除被认为不重要的提示缓存来缓解此问题,通常依赖估算的重要性评分。近期工作提出“窥视未来”机制,利用草稿生成器生成近似未来响应,从而更准确评估缓存重要性。但该方法需昂贵的草稿生成,带来显著预填充开销,限制实际应用。为此,我们提出LookaheadKV,一种轻量级淘汰框架,可在不显式生成草稿的前提下利用未来响应的优势。通过在Transformer层中加入参数高效的模块,训练其高精度预测真实重要性得分。该设计实现与简单启发式方法相当的运行时开销,同时优于更复杂的近似方法。在多种模型和长上下文理解基准上的实验表明,本方法不仅在各类任务中超越现有先进基线,且淘汰成本最高降低14.5倍,显著提升首个词生成速度。代码已开源:https://github.com/SamsungLabs/LookaheadKV。

原文摘要 · Abstract (English)

Transformer-based large language models (LLMs) rely on key-value (KV) caching to avoid redundant computation during autoregressive inference. While this mechanism greatly improves efficiency, the cache size grows linearly with the input sequence length, quickly becoming a bottleneck for long-context tasks. Existing solutions mitigate this problem by evicting prompt KV that are deemed unimportant, guided by estimated importance scores. Notably, a recent line of work proposes to improve eviction quality by "glimpsing into the future", in which a draft generator produces a surrogate future response approximating the target model's true response, and this surrogate is subsequently used to estimate the importance of cached KV more accurately. However, these approaches rely on computationally expensive draft generation, which introduces substantial prefilling overhead and limits their practicality in real-world deployment. To address this challenge, we propose LookaheadKV, a lightweight eviction framework that leverages the strength of surrogate future response without requiring explicit draft generation. LookaheadKV augments transformer layers with parameter-efficient modules trained to predict true importance scores with high accuracy. Our design ensures negligible runtime overhead comparable to existing inexpensive heuristics, while achieving accuracy superior to more costly approximation methods. Extensive experiments on long-context understanding benchmarks, across a wide range of models, demonstrate that our method not only outperforms recent competitive baselines in various long-context understanding tasks, but also reduces the eviction cost by up to 14.5x, leading to significantly faster time-to-first-token. Our code is available at https://github.com/SamsungLabs/LookaheadKV.

KV缓存长文本推理优化大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。