让大模型自己预测下一步操作，提前执行以减少等待时间。
SPORK: Self-Speculative Forking to Accelerate Agentic LLM Inference
- 大模型自预测下一步工具调用，无需额外训练或历史数据。
- 实测使Qwen3-32B推理速度提升18%，延迟从131.9秒降至108.1秒。
- 无需重训练、无额外模型，可直接部署在标准API上。
LLM代理在科研、编程和问答中日益普及，但其思维-行动-观察循环常为串行：模型推理后发出工具调用，需等待结果返回，期间GPU空闲。我们发现，该等待占总时间的16%-37%（本工作）或35%-61%（先前报告）。现有推测执行方法依赖辅助预测器、历史轨迹或静态流程图，难以实现零训练、即插即用。本文提出SPORK（Self-sPeculative fORKing），利用模型自身作为预测器——在生成起始处分叉探测，对Qwen3-32B的未来工具名预测准确率达74.6%-99.6%。控制器提前调度推测的工具调用，与后续思维解码并行。成本模型确定推测何时盈亏，各组件分别优化对应项：前缀缓存分叉降低探测开销，置信度门过滤误判，部分标记接受将被拒绝的推测转为推测解码草稿。成功则工具结果在推理结束时就绪；失败则回退至串行执行，不损失正确性。在真实工具基准上，SPORK将Qwen3-32B的GAIA P95降低18%（131.9→108.1秒），且在4B至32B模型规模、密集与混合专家模型上均有效，任务准确率与基线相差不超过1个百分点。SPORK作为轻量控制器部署于标准补全API之上（无需重训练、无辅助模型、无离线轨迹），并与令牌级推测解码正交。代码已开源：https://github.com/baihuajun24/spork。
原文摘要 · Abstract (English)
LLM agents are becoming a common interface for research, coding, and question answering, yet their Thought-Action-Observation loop is often serial: the model reasons, emits a tool call, then idles the GPU until the result returns. This wait consumes 16-37% of wall time in our workloads and 35-61% in prior reports. Speculative tool execution can hide this wait, but existing systems need auxiliary predictors, historical traces, or static workflow graphs, leaving a gap for training-free, day-one deployment. We observe that the model can be its own predictor: a probe forked at the start of generation predicts Qwen3-32B's upcoming tool name with 74.6-99.6% accuracy across five benchmarks. We present SPORK (Self-sPeculative fORKing), a training-free controller that dispatches the speculated tool call early, overlapping its execution with the remaining chain-of-thought decode. A cost model captures when speculation breaks even, and each component improves one of its terms: a prefix-cache fork cuts probe cost, a confidence gate filters mispredictions, and partial-token accept turns rejected probes into speculative-decoding drafts. On acceptance, the tool result is ready when reasoning ends; on rejection, SPORK falls back to serial execution with no correctness penalty. On real-tool benchmarks, SPORK cuts Qwen3-32B's GAIA P95 by 18% (131.9 to 108.1 s); the mechanism holds across model sizes from 4B to 32B and across dense and mixture-of-experts models, with task accuracy within 1 pp of baseline or better wherever measured. SPORK deploys as a thin controller over standard completion APIs (no retraining, no auxiliary models, no offline traces) and is orthogonal to token-level speculative decoding. SPORK is open source at https://github.com/baihuajun24/spork.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。