通过推测性调用工具提升智能体推理速度
Optimizing Agentic Language Model Inference via Speculative Tool Calls
- 用推测性方法提前预测并执行工具调用，减少等待延迟
- 实测推理吞吐量提升数百词每秒
- 适合需要高效调用外部工具的智能体系统开发者
语言模型（LM）日益依赖外部工具。基于LM的智能体框架常通过文件搜索、代码执行、API调用等与环境交互。现代基于推理的LM也利用网络搜索和Python代码执行来增强推理能力。尽管工具显著提升了LM能力，却也带来了推理过程中的性能瓶颈。本文提出新型系统优化，通过推测性工具调用并保持执行序列驻留推理引擎，以最小化开销。该优化使托管LM智能体推理时的吞吐量提升数百词每秒。我们对算法进行理论分析，揭示最优推测配置；并建议新增“工具缓存”API端点，便于LM提供商快速集成此优化。
原文摘要 · Abstract (English)
Language models (LMs) are becoming increasingly dependent on external tools. LM-based agentic frameworks frequently interact with their environment via such tools to search files, run code, call APIs, etc. Further, modern reasoning-based LMs use tools such as web search and Python code execution to enhance their reasoning capabilities. While tools greatly improve the capabilities of LMs, they also introduce performance bottlenecks during the inference process. In this paper, we introduce novel systems optimizations to address such performance bottlenecks by speculating tool calls and forcing sequences to remain resident in the inference engine to minimize overheads. Our optimizations lead to throughput improvements of several hundred tokens per second when hosting inference for LM agents. We provide a theoretical analysis of our algorithms to provide insights into speculation configurations that will yield the best performance. Further, we recommend a new "tool cache" API endpoint to enable LM providers to easily adopt these optimizations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。