通过预判工具调用，让大模型生成与工具执行并行，显著降低智能体响应延迟。
Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving
- 预判重复任务模式，提前并行执行工具调用
- 平均任务完成时间减少43.5%，工具延迟降低1.8倍
- 适合高并发、低延迟的智能体服务场景
基于大模型的智能体通过模型生成与工具执行的串行循环完成任务。当前的服务系统将此循环序列化，导致工具延迟暴露在任务关键路径上。本文提出PASTE——一种感知工具的智能体服务系统，能够从重复的智能体行为模式中预测未来的具体工具调用，并在大模型生成期间进行推测性执行。PASTE将推测结果隔离，直至被大模型确认，同时联合调度工具执行与大模型会话返回，避免瓶颈转移到GPU。在深度研究、代码生成和科学智能体等多种工作负载下，PASTE将平均任务完成时间降低43.5%，工具延迟降低1.8倍。
原文摘要 · Abstract (English)
LLM-powered agents execute tasks through a sequential loop of model generation and tool execution. Today's serving systems serialize this loop, leaving tool latency exposed on the task critical path. This paper presents PASTE, a tool-aware agent-serving system that predicts concrete future tool invocations from recurring agent patterns and executes them speculatively while the LLM is still generating. PASTE isolates speculative results until confirmed by the LLM and jointly schedules tool execution and returning LLM sessions to avoid shifting bottlenecks to the GPU. Across deep research, coding, and scientific-agent workloads, PASTE reduces average task completion time by 43.5% and lowers observed tool latency by 1.8x.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。