用树搜索提升数学推理，发现奖励模型反而拖后腿。
Limits of PRM-Guided Tree Search for Mathematical Reasoning with LLMs
- 用自适应算法探索多条解题路径，尝试优化奖励模型评分
- 树搜索未比传统方法更好，且成本更高，效果不显著
- 奖励模型深度越深越不可靠，外推能力差，不适合复杂推理
尽管基于思维链的 Best-of-N 选择在大语言模型数学推理中流行，但其线性结构无法捕捉复杂问题求解中的分支与探索特性。本文提出一种自适应算法，在难以计算的动作空间上最大化过程奖励模型（PRM）得分，并探究 PRM 引导的树搜索是否能通过探索多条部分解题路径提升数学推理能力。以 Qwen2.5-Math-7B-Instruct 及其配套的 PRM 为案例，在 23 个多样化数学问题上发现：(1) 尽管成本更高，PRM 引导的树搜索相比 BoN 无统计显著改进；(2) 蒙特卡洛树搜索和束搜索优于其他 PRM 引导方法；(3) PRM 对状态价值的近似能力差，且随着推理深度增加可靠性下降；(4) PRM 在分布外泛化性能差。这一表现源于树搜索对不可靠的 PRM 分数依赖更强，表明在树搜索有效提升数学推理前，需发展更可靠的奖励建模方式。
原文摘要 · Abstract (English)
While chain-of-thought prompting with Best-of-N (BoN) selection has become popular for mathematical reasoning in large language models (LLMs), its linear structure fails to capture the branching and exploratory nature of complex problem-solving. In this work, we propose an adaptive algorithm to maximize process reward model (PRM) scores over the intractable action space, and investigate whether PRM-guided tree search can improve mathematical reasoning by exploring multiple partial solution paths. Across $23$ diverse mathematical problems using Qwen2.5-Math-7B-Instruct with its associated PRM as a case study, we find that: (1) PRM-guided tree search shows no statistically significant improvements over BoN despite higher costs, (2) Monte Carlo tree search and beam search outperform other PRM-guided tree search methods, (3) PRMs poorly approximate state values and their reliability degrades with reasoning depth, and (4) PRMs generalize poorly out of distribution. This underperformance stems from tree search's greater reliance on unreliable PRM scores, suggesting different reward modeling is necessary before tree search can effectively enhance mathematical reasoning in LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。