通过智能预判和历史复用，让大模型调用工具快4.2倍。
ToolSpec: Accelerating Tool Calling via Schema-Aware and Retrieval-Augmented Speculative Decoding
- 基于工具模式和状态机生成精准预判草案。
- 在多个基准上实现最高4.2倍的调用加速。
- 无需训练，可直接接入现有大模型系统。
工具调用显著提升了大语言模型的实际应用能力，使其能够与外部应用交互。随着模型能力增强，复杂任务常需多步、多轮工具交互，导致显著延迟，影响实时服务。我们通过实证分析发现，工具调用序列具有高度结构化特征，遵循受限模式且重复性高。为此，提出ToolSpec：一种面向工具模式、结合检索增强的推测解码方法。该方法利用预定义工具模式生成准确草案，通过有限状态机在确定性填充与推测生成间交替处理固定与可变字段；同时，检索并复用相似历史调用作为草案，进一步提升效率。ToolSpec为即插即用方案，可无缝集成至现有大模型流程。在多个基准测试中，其速度提升最高达4.2倍，显著优于现有无训练推测解码方法。
原文摘要 · Abstract (English)
Tool calling has greatly expanded the practical utility of large language models (LLMs) by enabling them to interact with external applications. As LLM capabilities advance, effective tool use increasingly involves multi-step, multi-turn interactions to solve complex tasks. However, the resulting growth in tool interactions incurs substantial latency, posing a key challenge for real-time LLM serving. Through empirical analysis, we find that tool-calling traces are highly structured, conform to constrained schemas, and often exhibit recurring invocation patterns. Motivated by this, we propose ToolSpec, a schema-aware, retrieval-augmented speculative decoding method for accelerating tool calling. ToolSpec exploits predefined tool schemas to generate accurate drafts, using a finite-state machine to alternate between deterministic schema token filling and speculative generation for variable fields. In addition, ToolSpec retrieves similar historical tool invocations and reuses them as drafts to further improve efficiency. ToolSpec presents a plug-and-play solution that can be seamlessly integrated into existing LLM workflows. Experiments across multiple benchmarks demonstrate that ToolSpec achieves up to a 4.2x speedup, substantially outperforming existing training-free speculative decoding methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。