arXiv:2607.23933cs.DCcs.AI2026-07

通过预加载沙箱降低大模型代理延迟,兼顾资源与响应速度。

SpecBox: Speculative Sandbox Scheduling for Efficient LLM Agent Serving

论文配图:SpecBox: Speculative Sandbox Scheduling for Efficient LLM Agent Serving
图 1 · 摘自论文原文
  • 根据关键词和语义匹配提前预热沙箱,与推理并行。
  • 在高并发多轮任务中,尾部延迟降低2.9倍,内存消耗减少45.9%。
  • 适合需要低延迟、高并发的智能代理部署场景。

随着大模型代理越来越多地依赖模型上下文协议(MCP)调用隔离外部沙箱,分布式沙箱部署带来了资源利用率与交互尾延迟之间的根本矛盾。长期保留沙箱导致大规模内存开销,而按需启动则带来严重的冷启动问题,影响多租户、多轮次代理工作负载的响应性能。为此,我们提出SpecBox,一种面向动态大模型代理执行流水线的运行时系统。其核心采用关键词匹配与流式语义嵌入,实现意图驱动的沙箱预热,在大模型生成令牌过程中识别待执行工具需求,并将沙箱初始化完全重叠于模型推理阶段。为延长预热窗口,系统基于沙箱依赖图构建上下文感知的随机预取机制,可概率性预测未来沙箱切换。此外,还引入两个正交优化:语义结果缓存以消除重复沙箱调用,以及专用的带外共享内存传输通道,绕过传统网络序列化实现零拷贝数据传递。在高并发多轮代理日志上的评估显示,SpecBox相较按需沙箱基线,端到端尾延迟最高降低2.9倍,相比永久保留沙箱方案,峰值内存消耗降低45.9%。

原文摘要 · Abstract (English)

As LLM agents increasingly rely on the Model Context Protocol (MCP) to invoke isolated external sandboxes, disaggregated sandbox deployment introduces a fundamental tension between resource utilization and interactive tail latency. Persistent long-lived sandbox reservations incur excessive memory overhead at scale, while lazy on-demand instantiation generates severe cold-start penalties that degrade response performance under multi-tenant, multi-turn agent workloads. To resolve this dilemma, we present SpecBox, a runtime built around speculative sandbox preallocation tailored for dynamic LLM agent execution pipelines. At its core, SpecBox implements keyword matching and streaming semantic embedding to enable intent-driven sandbox prewarming, which identifies pending tool execution demands mid-LLM token generation and fully overlaps sandbox bootstrapping with model inference. To extend prewarming windows across sequential agent steps, the framework leverages context-aware stochastic prefetching atop a sandbox dependency graph to probabilistically forecast future sandbox switches ahead of execution. We complement these speculative mechanisms with two orthogonal optimizations: a semantic result cache that prunes redundant repeated sandbox invocations, and a dedicated out-of-band shared-memory transport plane that bypasses conventional network serialization to deliver zero-copy artifact transfers. Evaluated on high-concurrency multi-turn agent traces, our prototype demonstrates that SpecBox cuts P99 end-to-end latency by up to $2.9\times$ relative to the on-demand sandbox baseline, while slashing peak memory consumption by $45.9\%$ compared to permanently reserved sandbox deployments.

大模型代理沙箱调度延迟优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。