arXiv:2608.00814cs.CL2026-08

让大模型提前并行预测工具调用参数,提速近4倍。

OoO-Spec: Out-of-Order Semantic Speculation for Fast Tool Calling

论文配图:OoO-Spec: Out-of-Order Semantic Speculation for Fast Tool Calling
图 1 · 摘自论文原文
  • 并行预测函数与参数,不等逐字生成
  • 实测比传统方法快2.46至5.34倍,平均3.89倍
  • 一个轻量侧车模型适配多类大模型,无需重训

LLM 生成工具调用时逐词进行,尽管函数选择和参数值可从请求与工具模式中并行预测。ToolSpec 通过草稿模式令牌和检索历史调用降低开销,但无法提出请求相关的缺失值。我们提出 OoO-Spec,可在请求到达时并行计算这些缺失语义。在请求到达时,一个 Qwen3-0.6B 侧车模型一次性并行预测函数选择和所有模式定义的参数槽位,同时目标模型开始 ToolSpec 解码。运行时合并槽位值,生成调用文本,并提供给后续候选构建轮次。目标模型非阻塞轮询,使用自身分词器重新分词准备提示,且保留唯一验证与提交权限。侧车模型仅用一次 LoRA 训练于 Qwen2.5-32B 教师轨迹,在 Qwen2.5、Qwen3 及 Llama 目标上无需目标特定训练。在七个全排名目标和三个基准测试下,贪婪批处理单解码中,OoO-Spec 在全部 21 个目标-基准组合中均最快,比自回归解码快 2.46×–5.34×,平均 3.89×,优于 ToolSpec 的 2.95×。在所有可比组合中也超越每个已发布学习草稿器。在 Qwen3-4B、8B、14B、32B 目标上,该侧车模型平均使 ToolSpec 提升 34.1%。其紧凑语义负载每请求约 85 字节(不含协议元数据),支持有效分 GPU 重叠。

原文摘要 · Abstract (English)

LLMs generate tool calls token by token, even though the function choice and argument values can often be predicted in parallel from the request and tool schema. ToolSpec reduces this cost by drafting schema tokens and retrieving earlier calls, but cannot propose request-specific values absent from either source. We present OoO-Spec, which computes these missing semantics out of order. At request arrival, a Qwen3-0.6B sidecar predicts the function choice and all schema-defined argument slots in one parallel request-level wave while the target begins ToolSpec decoding. The runtime joins the slot values, renders the resulting call as text, and exposes it to subsequent candidate-construction rounds. The target polls without blocking, re-tokenizes a ready hint with its own tokenizer, and remains the sole verifier and commit authority. The sidecar is trained once with LoRA on Qwen2.5-32B teacher traces and used unchanged across Qwen2.5, Qwen3, and Llama targets, without target-specific drafter training. Across seven fully ranked targets and three benchmarks under greedy batch-one decoding, OoO-Spec is fastest among all evaluated methods in all 21 target-benchmark cells, reaching 2.46x-5.34x over autoregressive decoding with an unweighted mean of 3.89x, versus 2.95x for ToolSpec. It also outperforms every evaluated released learned drafter in each comparable cell. Across Qwen3-4B, 8B, 14B, and 32B targets, the same sidecar improves on ToolSpec by 34.1% on average. Its compact semantic payload averages 85 bytes per request excluding protocol metadata, supporting effective split-GPU overlap.

工具调用推理加速并行生成大模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。