arXiv:2608.14624cs.AI2026-08

让大模型服务提前预测并复用代理执行的缓存,提升推理速度。

Learning Agent Execution for KV-Cache Management in Agentic Serving

论文配图:Learning Agent Execution for KV-Cache Management in Agentic Serving
图 1 · 摘自论文原文
  • 基于代理执行行为在线学习缓存复用规律,主动预加载和淘汰。
  • 缓存命中率提升10-18个百分点,首字延迟降低18%-45%。
  • 无需预先定义流程,适用于真实多代理场景,适合高并发服务优化。

多代理大模型系统已成为AI服务的重要部署范式,每个用户请求被分解为一系列专用代理的执行序列。在这些工作流中,每个代理反复执行包含系统提示、工具定义和少量示例的固定上下文,为KV缓存复用提供了巨大机会。然而,现有大模型服务系统仅通过前缀缓存和基于访问频率的替换策略管理缓存,导致可复用的代理上下文在下次调用前就被淘汰,造成重复计算。我们提出CacheScout,一种面向多代理大模型服务的代理感知型KV缓存运行时层。核心洞察是:未来的缓存复用由代理执行语义决定,而非仅依赖缓存新近度。CacheScout通过在线学习代理执行转移行为,无需预设工作流图或离线训练,利用学习到的执行模型指导缓存淘汰与主动预取,同时不改变服务关键路径。我们在vLLM上实现CacheScout。在典型真实多代理负载下,缓存命中率提升10-18个百分点,平均首令牌时间(TTFT)降低18-45%,每轮平均延迟降低29-38%,峰值吞吐量提升最高达57%。该优势也推广至更大模型,在减少TTFT最高达54%的同时,维持37%更高吞吐量。

原文摘要 · Abstract (English)

Multi-agent LLM systems have emerged as an important deployment paradigm for AI services, where each user request is decomposed into a sequence of specialized agents. Across these workflows, every agent repeatedly executes a fixed context consisting of system prompts, tool definitions, and few-shot examples, creating substantial opportunities for KV-cache reuse. Existing LLM serving systems, however, manage KV-cache reactively using prefix caching and recency-based replacement, causing reusable agent contexts to be evicted before their next invocation and forcing repeated recomputation. We present CacheScout, an agent-aware KV-cache runtime layer for multi-agent LLM serving. The key insight is that future KV-cache reuse is governed by agent execution semantics rather than cache recency alone. CacheScout captures these semantics by learning agent execution transitions online, without requiring predefined workflow graphs or offline training, and uses the learned execution model to guide both cache eviction and proactive prefetching while leaving the serving critical path unchanged. We implement CacheScout on top of vLLM. Across representative real-world multi-agent workloads, CacheScout improves KV-cache hit rate by 10-18 percentage points, reduces mean TTFT by 18-45%, lowers mean per-turn latency by 29-38%, and increases peak throughput by up to 57%. These benefits also generalize to larger models, reducing TTFT by up to 54% while sustaining 37% higher throughput.

大模型服务缓存优化多代理系统推理加速

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。