用大模型自己判断哪些记忆该删，省内存不丢推理能力
SideQuest: Model-Driven KV Cache Management for Long-Horizon Agentic Reasoning
- 让大模型自检上下文，动态压缩无用的键值缓存
- 在215个样本训练下，内存峰值降低65%且准确率几乎不变
- 适合长时推理任务，如多步搜索、深度研究等场景
长时间运行的智能体任务（如深度调研）需跨多个网页和文档进行多跳推理。此类任务中，外部检索的文本占上下文主导，导致内存快速增长并限制解码性能。现有基于启发式的键值缓存压缩方法难以有效支持多步推理模型。本文提出SideQuest——一种利用大推理模型（LRM）自身判断上下文令牌有用性的新型压缩方法。为避免管理过程污染模型记忆，将缓存压缩设为与主推理并行的辅助任务。评估显示，仅用215个样本训练的模型，便可在智能体任务中将峰值令牌使用量减少高达65%，同时保持精度几乎不变，显著优于传统启发式压缩方法。
原文摘要 · Abstract (English)
Long-running agentic tasks, such as deep research, require multi-hop reasoning over information distributed across multiple webpages and documents. In such tasks, the LLM context is dominated by tokens from external retrieval, causing memory usage to grow rapidly and limiting decode performance. While several KV cache compression techniques exist for long-context inputs, we find that existing heuristics fail to support multi-step reasoning models effectively. We address this challenge with SideQuest -- a novel approach that leverages the Large Reasoning Model (LRM) itself to perform KV cache compression by reasoning about the usefulness of tokens in its context. To prevent the tokens associated with this management process from polluting the model's memory, we frame KV cache compression as an auxiliary task executed in parallel to the main reasoning task. Our evaluations, using a model trained with just 215 samples, show that SideQuest reduces peak token usage by up to 65% on agentic tasks with minimal degradation in accuracy, outperforming heuristic-based KV cache compression techniques.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。