通过不确定性引导探索，提升多轮智能体强化学习的稳定性与效率
T$^2$PO: Uncertainty-Guided Exploration Control for Stable Multi-Turn Agentic Reinforcement Learning
- 在词和回合层级动态监控不确定性，触发思考干预或重采样
- 在WebShop等环境上显著减少训练崩溃，提升任务完成率
- 适合研究多轮推理、强化学习稳定性的研究人员使用
近期多轮强化学习进展显著提升了大模型在复杂交互任务中的表现。尽管已有细粒度信用分配和轨迹过滤等稳定技术，训练不稳定仍普遍存在，常导致训练崩溃。本文认为，问题根源在于多轮设置下探索效率低下，策略持续生成低信息动作，无法降低不确定性或推动任务进展。为此，提出词级与回合级策略优化（T²PO），一种基于不确定性的探索控制框架。在词层级，监测不确定性变化，当边际变化低于阈值时触发思考干预；在回合层级，识别探索进展微弱的交互，动态重采样以避免无效回滚。在WebShop、ALFWorld和Search QA等多个环境中评估，T²PO显著提升训练稳定性与性能，实现更高效的探索。代码已开源：https://github.com/WillDreamer/T2PO。
原文摘要 · Abstract (English)
Recent progress in multi-turn reinforcement learning (RL) has significantly improved reasoning LLMs' performances on complex interactive tasks. Despite advances in stabilization techniques such as fine-grained credit assignment and trajectory filtering, instability remains pervasive and often leads to training collapse. We argue that this instability stems from inefficient exploration in multi-turn settings, where policies continue to generate low-information actions that neither reduce uncertainty nor advance task progress. To address this issue, we propose Token- and Turn-level Policy Optimization (T$^2$PO), an uncertainty-aware framework that explicitly controls exploration at fine-grained levels. At the token level, T$^2$PO monitors uncertainty dynamics and triggers a thinking intervention once the marginal uncertainty change falls below a threshold. At the turn level, T$^2$PO identifies interactions with negligible exploration progress and dynamically resamples such turns to avoid wasted rollouts. We evaluate T$^2$PO in diverse environments, including WebShop, ALFWorld, and Search QA, demonstrating substantial gains in training stability and performance improvements with better exploration efficiency. Code is available at: https://github.com/WillDreamer/T2PO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。