针对大模型代理的在线键值缓存压缩，提出有效减少内存占用的方法。
Practical Online KV Cache Compaction for LLM Agents: An Empirical Study
- 采用延迟压缩策略，利用未来查询作为代理，实现在线缓存优化。
- 在不同模型上实现80%缓存减少，同时保持大部分推理准确率。
- 推荐使用令牌淘汰法，比注意力匹配更适用于代理不精确场景。
大模型代理在推理过程中积累长序列的思维步骤、工具调用和环境反馈，导致键值（KV）缓存成为主要的推理瓶颈。尽管缓存压缩可缓解此问题，但以往方法多假设上下文静态，未来查询已知或可离线近似。而代理需要在线压缩：在无法预知未来相关性时，必须在新信息到来前压缩，依赖低成本的代理查询以适应推理路径。本文研究了在令牌淘汰（TE）与注意力匹配（AM）两种策略下，如何适配代理查询（如边界查询、重复预填充查询、延迟未来生成查询）进行在线压缩。在BrowseComp-Plus与WideSearch数据集上的实验表明，立即压缩通常损害性能，而延迟压缩至使用代理未来查询可显著恢复性能差距。此外，在代理不准确时，令牌淘汰法往往比注意力匹配更鲁棒。跨不同规模模型的实验显示，采用令牌淘汰法可在保持绝大多数准确率的同时，将KV缓存降低80%，并提升吞吐量，优于无压缩基线。这些结果凸显代理查询选择是实用在线压缩的核心设计因素。
原文摘要 · Abstract (English)
LLM agents accumulate long trajectories of reasoning steps, tool calls, and environment feedback, making the KV cache a major inference bottleneck. KV cache compaction can reduce this cost, but most prior methods assume a static context where future queries are known or can be approximated offline. Agents instead require online compaction: new information must be compressed before future relevance is known, using proxy queries cheap enough for the inference path. We study online compaction across token eviction (TE) and attention matching (AM), adapting both to compact agent turns and comparing cheap proxy sources such as boundary, repeat-prefill, and delayed future-generation queries. Experiments on BrowseComp-Plus and WideSearch show that immediate compaction often hurts performance, whereas delaying compaction to use the agent's future queries recovers much of the gap. Moreover, TE is often more robust than AM under imperfect proxies. Across models at different scales, TE preserves most of the accuracy while reducing KV cache by 80%, and can improve throughput over the no compaction baseline. These results position proxy-query selection as a core design choice for practical online KV compaction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。