arXiv:2511.02230cs.OScs.AI2025-11被引 44

通过缓存存活时间机制,提升多轮代理任务的推理效率与稳定性。

Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live

  • 为工具调用引入KV缓存存活时间,智能决定是否保留在显存中。
  • 在真实代理任务上平均任务完成时间提升8倍以上,吞吐量显著改善。
  • 适用于对延迟敏感的多轮智能体系统,尤其适合工具调用频繁的场景。

KV缓存管理对高效大模型推理至关重要。现有推理引擎在新请求等待时会清除已完成请求的KV缓存以提高利用率,但这一策略在代理类工作负载中失效——因代理任务在大模型调用与工具调用间交替进行,产生暂停,阻碍跨轮次的缓存复用。由于许多工具调用持续时间远短于人类响应的多轮对话,保留其对应的KV缓存具有潜力。然而仍存在挑战:既要权衡重计算或卸载的成本,也要应对工具调用时长不可预测带来的风险。本文提出Continuum,一种面向多轮代理任务的推理系统,通过为KV缓存引入存活时间(TTL)机制来优化任务完成时间。对于生成工具调用的请求,Continuum根据重加载成本和因淘汰导致的排队延迟,选择性地将KV缓存固定在显存中。当TTL到期后,缓存自动被释放以腾出显存,保证在边缘情况下的鲁棒性。结合程序级先到先服务策略,Continuum维持了多轮连续性,有效降低代理工作流延迟。在真实代理任务(SWE-Bench、BFCL、OpenHand)上,使用Llama-3.1 8B/70B、Gemma-3 12B、GLM-4.5 355B进行评估,结果显示平均任务完成时间提升超过8倍,同时显著提高吞吐量。

原文摘要 · Abstract (English)

KV cache management is essential for efficient LLM inference. To maximize utilization, existing inference engines evict finished requests' KV cache if new requests are waiting. This policy breaks for agentic workloads, which interleave LLM calls with tools, introducing pauses that prevent effective KV reuse across turns. Since many tool calls have much shorter durations than human response multi-turn chatbot, it would be promising to retain the KV cache in during these tools. However, many challenges remain. First, we need to consider both the potential cost of recomputation or reloading (if offloading enabled) as well as the increasing queueing delays after eviction from GPU. Second, due to the internal variance of tool call durations, the method needs to remain robust under limited predictability of tool call durations. We present Continuum, a serving system to optimize job completion time for multi-turn agent workloads by introducing time-to-live mechanism for KV cache retention. For requests that generate tool calls, Continuum selectively pins the KV cache in GPU memory with a time-to-live value determined by the reload cost and potential queueing delay induced by eviction. When the TTL expires, the KV cache can be automatically evicted to free up GPU memory, providing robust performance under edge cases. When combined with program-level first-come-first-serve, Continuum preserves multi-turn continuity, and reduces delay for agentic workflows. Evaluations on real-world agents (SWE-Bench, BFCL, OpenHand) with Llama-3.1 8B/70B, Gemma-3 12B, and GLM-4.5 355B shows that Continuum improves the average job completion times by over 8x while improving throughput.

大模型推理缓存管理智能体系统性能优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。