用树形搜索提升LLM智能体强化学习的效率与监督信号
Tree Search for LLM Agent Reinforcement Learning
- 构建树状轨迹结构，共享前缀以提升采样效率
- 在仅使用最终奖励下实现每步过程监督，提升训练信号密度
- 适用于长周期多轮任务，尤其适合需要精细决策的智能体
大语言模型智能体的强化学习近期取得显著进展。在长期、多轮任务中，仅依赖结果奖励的方法常面临稀疏监督问题。为此，我们提出基于树搜索的分组相对策略优化（Tree-GRPO），其中每个树节点代表完整的智能体交互步骤。通过共享公共前缀，树搜索采样可在固定令牌或工具调用预算内增加回溯次数。此外，我们发现树状轨迹天然支持仅用最终奖励构建逐步过程监督信号。基于此，Tree-GRPO 在树内与树间两个层面估计分组相对优势。理论分析表明，树内分组相对策略优化的目标等价于步骤级直接偏好学习。在11个数据集和3类问答任务上的实验表明，所提出的树基强化学习方法优于链式强化学习方法。
原文摘要 · Abstract (English)
Recent advances in reinforcement learning (RL) have significantly enhanced the agentic capabilities of large language models (LLMs). In long-term and multi-turn agent tasks, existing approaches driven solely by outcome rewards often suffer from the problem of sparse supervision. To address the challenge, we propose Tree-based Group Relative Policy Optimization (Tree-GRPO), a grouped agent RL method based on tree search, where each tree node represents the complete agent interaction step. By sharing common prefixes, the tree search sampling increases the number of rollouts achievable within a fixed budget of tokens or tool calls. Moreover, we find that the tree-structured trajectory naturally allows the construction of step-wise process supervised signals even using only the outcome reward. Based on this, Tree-GRPO estimates the grouped relative advantages both on intra-tree and inter-tree levels. Through theoretical analysis, we demonstrate that the objective of intra-tree level group relative policy optimization is equivalent to that of step-level direct preference learning. Experiments across 11 datasets and 3 types of QA tasks demonstrate the superiority of the proposed tree-based RL over the chain-based RL method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。