通过推测性探索突破思维树推理的效率瓶颈,实现1.2至4.1倍加速。
Breaking the Reward Barrier: Accelerating Tree-of-Thought Reasoning via Speculative Exploration

- 采用推测性路径选择、动态资源分配与自适应提前终止,打破奖励同步限制。
- 在多种思维树算法和模型上实现1.2~3倍速度提升,融合令牌级推测后达4.1倍。
- 适合需要高效复杂推理的场景,如数学证明与编程任务,尤其对大规模模型有意义。
思维树(ToT)将大语言模型推理建模为基于树的搜索,在解决复杂数学与编程任务方面展现出巨大潜力。然而,其效率受限于奖励依赖屏障——由顺序奖励引导探索引发的同步瓶颈,制约了搜索并行性并引入显著延迟。现有系统优化主要针对线性思维链(CoT),无法解决此问题,导致思维树效率长期未被充分挖掘。为此,本文提出SPEX,通过推测性探索打破奖励同步障碍,引入三项关键技术:(i) 查询内推测路径选择,预测并扩展高潜力分支;(ii) 查询间预算分配,动态平衡跨查询的推测资源;(iii) 自适应早期终止,剪枝深度冗余分支以缓解搜索树偏斜。我们在SGLang框架上实现SPEX,评估涵盖多种ToT算法与LLM。实验表明,SPEX在不同算法下实现1.2~3倍加速;结合令牌级推测解码,累计提速最高达4.1倍。消融实验证实各技术贡献显著。SPEX标志着迈向高效可扩展思维树推理的重要一步,为大语言模型推理时扩展提供了必要的并行能力。
原文摘要 · Abstract (English)
Tree-of-Thought (ToT) reasoning structures Large Language Model (LLM) inference as a tree-based search, demonstrating strong potential for solving complex mathematical and programming tasks. However, its efficiency is constrained by the reward dependency barrier -- a synchronization bottleneck caused by sequential reward-guided exploration that limits search parallelism and introduces substantial latency. Prior system optimizations, mainly designed for linear Chain-of-Thought (CoT) reasoning, cannot address these challenges, leaving the efficiency of ToT underexplored. To enhance ToT reasoning efficiency, we observe that the reasoning paths can be explored speculatively to break the reward synchronization barrier. Therefore, in this paper, we propose SPEX and introduce three key techniques: (i) intra-query speculative path selection to predict and expand high-potential branches of ToT, (ii) inter-query budget allocation to balance speculative resource allocation across queries dynamically, and (iii) adaptive early termination to prune deep and redundant branches for a skewed search tree. We implement SPEX on top of the SGLang framework and evaluate it across diverse ToT algorithms and LLMs. Extensive experiments show that SPEX achieves $1.2 \sim 3 \times$ speedup for different ToT reasoning algorithms. Moreover, SPEX synergizes with token-level speculative decoding, achieving cumulative speedups of up to $4.1\times$. Ablation studies further confirm the contributions of each technique. Overall, SPEX represents a significant step toward efficient and scalable ToT reasoning, unlocking the parallelism required for high-performance inference-time scaling for LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。