用树搜索提升大模型强化学习，更高效探索推理路径。
TreeRL: LLM Reinforcement Learning with On-Policy Tree Search
- 直接在强化学习中引入策略树搜索，动态生成过程奖励。
- 在数学与编程推理任务上超越传统链式方法，性能显著提升。
- 无需额外训练奖励模型，且搜索效率更高，适合资源受限场景。
基于树搜索的强化学习在传统推理任务中表现优异。相比依赖结果监督的独立链采样策略，树搜索能更好探索推理空间，并在强化学习训练中提供密集的、基于策略的过程奖励，但在大模型的在线策略强化学习中仍研究不足。本文提出TreeRL，一种直接将在线策略树搜索融入强化学习训练的框架。该方法引入中间层监督，无需单独训练奖励模型，避免了分布不匹配和奖励欺骗问题。我们还设计了一种低成本的树搜索策略，通过在高不确定性中间步骤进行有策略的分支，而非随机分支，在相同生成词元预算下实现更高的搜索效率。在多个具有挑战性的数学与代码推理基准测试中，TreeRL的表现显著优于传统ChainRL，展现了树搜索在大模型中的巨大潜力。代码已开源：https://github.com/THUDM/TreeRL。
原文摘要 · Abstract (English)
Reinforcement learning (RL) with tree search has demonstrated superior performance in traditional reasoning tasks. Compared to conventional independent chain sampling strategies with outcome supervision, tree search enables better exploration of the reasoning space and provides dense, on-policy process rewards during RL training but remains under-explored in On-Policy LLM RL. We propose TreeRL, a reinforcement learning framework that directly incorporates on-policy tree search for RL training. Our approach includes intermediate supervision and eliminates the need for a separate reward model training. Existing approaches typically train a separate process reward model, which can suffer from distribution mismatch and reward hacking. We also introduce a cost-effective tree search approach that achieves higher search efficiency under the same generation token budget by strategically branching from high-uncertainty intermediate steps rather than using random branching. Experiments on challenging math and code reasoning benchmarks demonstrate that TreeRL achieves superior performance compared to traditional ChainRL, highlighting the potential of tree search for LLM. TreeRL is open-sourced at https://github.com/THUDM/TreeRL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。