提出新方法提升大模型智能体长任务决策能力,兼顾学习效率与公平性。
Progress- and Reliability-Oriented Group Policy Optimization for Agentic Reinforcement Learning

- 通过保持动作前缀一致,实现更精准的步骤级策略优化
- 在ALFWorld和WebShop上优于现有基线,计算开销相近
- 适合需要稳定长程决策的大模型智能体研究者
基于群体的强化学习已成为提升大语言模型智能体在长周期交互任务中表现的有效范式。为获得比轨迹级优化更细粒度的策略更新,近期工作转向步骤级群体强化学习,将回溯批次中的中间步骤分组并进行比较。然而,步骤级优势估计对分组方式敏感:按宽泛状态键分组虽提升覆盖范围,但可能比较不同历史背景下的动作;强制历史一致性虽保证比较公平,却导致分组碎片化并丢失同伴比较信号。本文提出无学习评论器的ProGPO(进度与可靠性导向群体策略优化)方法,实现上下文一致的步骤级学习。ProGPO保留精确前缀动作对比,并用基于回溯的状态势能推导的转移信用来补充稀疏的同伴比较。为可靠估计这些势能,ProGPO结合语义扩展与跨历史深度的逆方差融合。我们在ALFWorld和WebShop两个挑战性任务上,使用Qwen2.5-1.5B-Instruct进行评估,结果表明ProGPO在相当计算开销下优于匹配的代理强化学习基线;额外的Qwen2.5-3B-Instruct实验进一步验证了该方法的可扩展性。
原文摘要 · Abstract (English)
Group-based reinforcement learning (RL) has become an effective paradigm for improving large language model agents on long-horizon interactive tasks. To obtain finer-grained policy updates than trajectory-level optimization, recent work has moved toward step-level group-based RL, where intermediate steps are grouped and compared within a rollout batch. However, step-level advantage estimation is sensitive to how groups are formed: grouping by broad state keys improves coverage but may compare actions taken under different histories, while enforcing historical consistency yields fairer comparisons at the cost of fragmented groups and missing peer-comparison signal. In this paper, we propose ProGPO (Progress- and Reliability-Oriented Group Policy Optimization), a learned-critic-free method for context-consistent step-level learning. ProGPO keeps exact-prefix action comparison, and complements sparse peer comparisons with transition credit derived from rollout-based state potentials. To estimate these potentials reliably, ProGPO combines semantic expansion with inverse-variance fusion across history depths. We evaluate ProGPO on two challenging agentic tasks, ALFWorld and WebShop, with Qwen2.5-1.5B-Instruct. Results show that ProGPO improves over matched agentic RL baselines under comparable computational overhead, and additional Qwen2.5-3B-Instruct experiments further test the scalability of the proposed method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。