提升大模型推理速度与可解释性，让MCTS更准更快。
Interpretable Contrastive Monte Carlo Tree Search Reasoning
- 基于对比解码设计可解释奖励模型，增强推理透明度。
- 通过推测解码提速51.9%，在Blocksworld任务上比o1-mini高17.4%。
- 优化节点选择与反向传播策略，适合需要可解释推理的场景。
我们提出SC-MCTS*：一种面向大语言模型（LLMs）的新型蒙特卡洛树搜索（MCTS）推理算法，显著提升推理准确率与速度。现有MCTS方法常被忽视其相比思维链（CoT）速度慢的缺陷；且缺乏对组件的量化分析与可解释性研究；同时，奖励模型作为核心组件却极少被深入探讨。为此，我们进行了全面的消融实验与定量分析，揭示各组件对推理性能的影响。在此基础上，（i）设计基于对比解码的高可解释奖励模型；（ii）利用推测解码实现每节点51.9%的平均加速；（iii）改进UCT节点选择与反向传播机制。在Llama-3.1-70B模型下，于Blocksworld多步推理数据集上，相较o1-mini平均提升17.4%。代码已开源。
原文摘要 · Abstract (English)
We propose SC-MCTS*: a novel Monte Carlo Tree Search (MCTS) reasoning algorithm for Large Language Models (LLMs), significantly improves both reasoning accuracy and speed. Our motivation comes from: 1. Previous MCTS LLM reasoning works often overlooked its biggest drawback--slower speed compared to CoT; 2. Previous research mainly used MCTS as a tool for LLM reasoning on various tasks with limited quantitative analysis or ablation studies of its components from reasoning interpretability perspective. 3. The reward model is the most crucial component in MCTS, however previous work has rarely conducted in-depth study or improvement of MCTS's reward models. Thus, we conducted extensive ablation studies and quantitative analysis on components of MCTS, revealing the impact of each component on the MCTS reasoning performance of LLMs. Building on this, (i) we designed a highly interpretable reward model based on the principle of contrastive decoding and (ii) achieved an average speed improvement of 51.9% per node using speculative decoding. Additionally, (iii) we improved UCT node selection strategy and backpropagation used in previous works, resulting in significant performance improvement. We outperformed o1-mini by an average of 17.4% on the Blocksworld multi-step reasoning dataset using Llama-3.1-70B with SC-MCTS*. Our code is available at https://github.com/zitian-gao/SC-MCTS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。