提出分轮次的PPO算法，提升大模型多轮交互中的强化学习稳定性。
Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMs
- 在轮次级别构建马尔可夫决策过程，替代传统词元级建模
- 在WebShop和Sokoban任务上优于GRPO，长程推理效果更优
- 适合需要多轮对话与复杂推理的智能体训练场景
强化学习（RL）重新成为训练真实环境中交互式大语言模型（LLM）智能体的自然方法。然而，将广泛使用的组相对策略优化（GRPO）直接应用于多轮任务时，暴露了显著局限，尤其在需要长程推理的场景中。为解决这些问题，我们研究了更稳定有效的优势估计策略，特别是在多轮设置下。我们首先探索了近端策略优化（PPO）作为替代方案，发现其比GRPO更具鲁棒性。为进一步提升PPO在多轮场景下的表现，我们提出了转轮级PPO（turn-PPO），该方法基于轮次级别的马尔可夫决策过程（MDP），而非常见的词元级MDP。在WebShop和Sokoban数据集上的实验结果表明，turn-PPO在有无长程推理组件的情况下均表现出色。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has re-emerged as a natural approach for training interactive LLM agents in real-world environments. However, directly applying the widely used Group Relative Policy Optimization (GRPO) algorithm to multi-turn tasks exposes notable limitations, particularly in scenarios requiring long-horizon reasoning. To address these challenges, we investigate more stable and effective advantage estimation strategies, especially for multi-turn settings. We first explore Proximal Policy Optimization (PPO) as an alternative and find it to be more robust than GRPO. To further enhance PPO in multi-turn scenarios, we introduce turn-PPO, a variant that operates on a turn-level MDP formulation, as opposed to the commonly used token-level MDP. Our results on the WebShop and Sokoban datasets demonstrate the effectiveness of turn-PPO, both with and without long reasoning components.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。