AOSpec通过协同推测动作与观测，显著降低智能体服务延迟。
AOSpec: Action and Observation Co-Speculation for Low-Latency Agent Serving
- 协同推测动作和环境响应，打破串行执行瓶颈
- 平均端到端延迟降低11.8%-32.5%，尾部延迟降幅达42.8%
- 适用于高并发、低延迟场景，支持模型迁移无需重训
大型语言模型智能体越来越多地通过有状态工具执行任务，但模型生成与环境执行仍每步串行进行。随着解码加速，工具执行逐渐成为主要瓶颈。现有仅推测动作或仅推测观测的方法仍暴露大量延迟：关键性能集中在少数慢速调用，部分结果仅能通过执行揭示，且长时前瞻通常依赖越来越不可能的动作链预测。我们提出AOSpec，一种无损框架，实现智能体-环境全链路的动作与观测协同推测。期望值解码（EVD）引导观测推测聚焦于预期延迟收益最大的结果，优化隐藏时间而非命中率。对于仅执行可揭示的结果，AOSpec在隔离分支中提前启动关键延迟动作并隔离其影响；联合动作-状态验证（JASV）在复用前验证动作及其原始状态是否与已提交执行一致。JASV将长程动作依赖从全链预测重构为目标动作-状态验证，打破前瞻长度与准确率的权衡，实现长距离重叠而不破坏串行语义。在涵盖四个测试套件、五种智能体模型和五种服务速度的Terminal-Bench部署环境下，AOSpec优于所有实用基线，平均端到端延迟降低11.8%-32.5%，p99延迟最高减少42.8%。其性能随解码加速而提升，且观测模型可直接迁移至SWE-bench Verified无需重新训练。
原文摘要 · Abstract (English)
Large language model agents increasingly act through stateful tools, yet model generation and environment execution remain serialized at every step. As decoding accelerates, tool execution becomes a growing bottleneck. Existing action- or observation-only speculation leaves much of this latency exposed: value is concentrated in a few slow calls, some outcomes emerge only through execution, and longer lookahead typically requires an increasingly unlikely chain of action predictions. We present AOSpec, a lossless framework that co-speculates actions and observations across the full agent-environment loop. Expected Value Decoding (EVD) directs observation speculation toward outcomes with the greatest expected latency benefit, optimizing expected time hidden rather than hit rate. For outcomes only execution can reveal, AOSpec launches latency-critical target actions in isolated forks that contain their effects, while Joint Action-State Verification (JASV) verifies both the action and its origin state against committed execution before reuse. JASV recasts long-horizon action dependency from full-chain prediction into target action-state verification, breaking the lookahead--accuracy tradeoff and unlocking long-range overlap without sacrificing serial semantics. Across Terminal-Bench serving settings spanning four harnesses, five actor models, and five serving speeds, AOSpec outperforms every practical baseline, reducing mean end-to-end latency by 11.8-32.5% and p99 latency by up to 42.8%. Its gains increase as decoding accelerates, and its observation model transfers from Terminal-Bench to SWE-bench Verified without retraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。