通过状态化推理，显著降低多智能体工具调用的延迟。
Stateful Inference for Low-Latency Multi-Agent Tool Calling
- 保留对话状态，只处理新增内容，避免重复计算。
- 6轮任务中每轮提速2.1倍，35轮任务中位数轮次提速4.2倍。
- 适合高并发、长流程的多智能体系统部署使用。
多智能体工具调用正成为基于大模型系统的主流交互模式，但现有推理框架将每次工具调用视为独立请求，即使85%-95%的提示内容与前一轮相同，也需从头重新处理。本文提出一种状态化推理架构，将传统服务中每轮 $O(n_t)$ 的开销降至仅 $O(Δ_t)$ 的增量开销：持久化KV缓存跨轮次保留，并仅摄入新输入令牌；径向前缀缓存实现交错多智能体流量下的扩展；提示查找推测解码器加速结构化输出生成。在全新生成的工作负载上对比vLLM和SGLang，参考实现于6轮智能体工作流中每轮提速2.1倍，35轮任务中位数轮次提速4.2倍，端到端墙时长减半。性能优势源于状态复用与推测机制，而非单纯缓存。
原文摘要 · Abstract (English)
Multi-agent tool calling is becoming the dominant interaction pattern for LLM-based systems, yet existing inference frameworks treat each tool call as an independent request, re-processing the entire conversation from scratch even though 85-95% of the prompt is unchanged from the previous turn. We present a stateful inference architecture that converts the $O(n_t)$ per-turn cost of conventional serving into an $O(Δ_t)$ delta-only cost: a persistent KV cache lives across turns and advances by ingesting only the new tokens, while a radix prefix cache extends this across interleaved multi-agent traffic and a prompt-lookup speculative decoder accelerates structured output. Against vLLM and SGLang on novel, fully-generated workloads, the reference implementation is $2.1\times$ faster per turn on a 6-turn agentic workflow and $4.2\times$ on the median turn of a 35-turn one, halving end-to-end wall time. The advantage comes from stateful reuse and speculation, not caching.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。