通过预判查询优化缓存淘汰，提升长文本生成的效率与一致性。
Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query
- 用低成本伪查询预估真实推理时的注意力重要性，指导缓存淘汰。
- 在有限缓存预算下，LongBench指标提升1~4分，优于现有方法。
- 可与现有方法融合，适合追求高效部署的LLM应用开发者。
大语言模型依赖键值缓存（KV cache）加速解码，减少重复计算。但随着序列变长，缓存内存消耗急剧增加，影响部署效率。现有淘汰策略基于预填充阶段的注意力分数，与实际推理查询不一致，尤其在内存受限时表现不佳。本文提出前瞻查询缓存（Lookahead Q-Cache, LAQ），通过生成低成本的伪前瞻查询，更准确地逼近真实解码阶段的查询模式。利用这些前瞻查询作为重要性评估的观察窗口，LAQ实现了与实际推理场景更一致的缓存淘汰。在LongBench和Needle-in-a-Haystack基准测试中，LAQ在多种缓存预算下均优于现有方法，在有限缓存条件下实现1~4点的性能提升。此外，LAQ与现有方法具有互补性，可灵活组合以进一步优化效果。
原文摘要 · Abstract (English)
Large language models (LLMs) rely on key-value cache (KV cache) to accelerate decoding by reducing redundant computations. However, the KV cache memory usage grows substantially with longer text sequences, posing challenges for efficient deployment. Existing KV cache eviction methods prune tokens using prefilling-stage attention scores, causing inconsistency with actual inference queries, especially under tight memory budgets. In this paper, we propose Lookahead Q-Cache (LAQ), a novel eviction framework that generates low-cost pseudo lookahead queries to better approximate the true decoding-stage queries. By using these lookahead queries as the observation window for importance estimation, LAQ achieves more consistent and accurate KV cache eviction aligned with real inference scenarios. Experimental results on LongBench and Needle-in-a-Haystack benchmarks show that LAQ outperforms existing methods across various budget levels, achieving a 1 $\sim$ 4 point improvement on LongBench under limited cache budget. Moreover, LAQ is complementary to existing approaches and can be flexibly combined to yield further improvements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。