用树搜索提升大模型智能体的多轮决策能力，显著改善探索与奖励分配。
AT$^2$PO: Agentic Turn-based Policy Optimization via Tree Search
- 基于回合级树结构实现策略探索与奖励传播
- 在7个基准上平均性能提升最高达1.84个百分点
- 适合需要精细决策优化的多轮智能体研究
大型语言模型智能体通过交替进行内部推理与外部工具调用，已展现出处理多轮任务的强大能力。近年来，智能体强化学习作为关键的后训练范式受到广泛关注。本文提出AT²PO（基于树搜索的回合级智能体策略优化），一个统一的多轮智能体强化学习框架，解决了探索多样性不足、奖励稀疏分配和策略优化错位三大挑战。AT²PO引入回合级树结构，结合熵引导的树扩展以实现战略探索，以及回合级奖励分配以实现从稀疏结果中精细化传播奖励。同时，提出回合级策略优化目标，使策略更新与智能体交互的自然决策粒度对齐。AT²PO与树搜索正交，可无缝集成至任意多轮强化学习流程。在七个基准上的实验表明，其性能相较最先进基线平均提升最多1.84个百分点，消融实验证实了各组件的有效性。代码已公开于https://github.com/zzfoutofspace/ATPO。
原文摘要 · Abstract (English)
LLM agents have emerged as powerful systems for tackling multi-turn tasks by interleaving internal reasoning and external tool interactions. Agentic Reinforcement Learning has recently drawn significant research attention as a critical post-training paradigm to further refine these capabilities. In this paper, we present AT$^2$PO (Agentic Turn-based Policy Optimization via Tree Search), a unified framework for multi-turn agentic RL that addresses three core challenges: limited exploration diversity, sparse credit assignment, and misaligned policy optimization. AT$^2$PO introduces a turn-level tree structure that jointly enables Entropy-Guided Tree Expansion for strategic exploration and Turn-wise Credit Assignment for fine-grained reward propagation from sparse outcomes. Complementing this, we propose Agentic Turn-based Policy Optimization, a turn-level learning objective that aligns policy updates with the natural decision granularity of agentic interactions. ATPO is orthogonal to tree search and can be readily integrated into any multi-turn RL pipeline. Experiments across seven benchmarks demonstrate consistent improvements over the state-of-the-art baseline by up to 1.84 percentage points in average, with ablation studies validating the effectiveness of each component. Our code is available at https://github.com/zzfoutofspace/ATPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。