用图结构提升智能体强化学习的步骤级奖励分配精度
Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learning
- 构建状态转移图，利用全局信息衡量状态到目标距离
- 通过图上边的贡献度计算优势，实现细粒度奖励分配
- 在多个挑战性任务中显著提升训练效率与性能
群体强化学习方法在提升大语言模型性能方面取得显著进展，并迅速拓展至智能体任务。然而，其信用分配依赖于以最终结果为基础的粗粒度轨迹级归因，难以捕捉单个步骤的贡献，例如被失败轨迹掩盖的有价值步骤。为挖掘潜在信息并实现更真实的步骤级信用分配，我们提出图基群体策略优化（GraphGPO），首先将所有回溯轨迹聚合为统一的状态转移图，然后利用图中编码的全局信息估计每个状态到任务目标的距离。最后，基于该距离减少量，通过图基优势估计为每条边分配信用。实验表明，GraphGPO显著提升了训练效率，在多个挑战性基准测试中达到当前最优性能。
原文摘要 · Abstract (English)
Group-based reinforcement learning (RL) methods have achieved remarkable success in improving the performance of large language models (LLMs) and have been rapidly extended to agentic tasks. However, their credit assignment relies heavily on coarse-grained trajectory-level attribution according to final outcomes, making it difficult to capture the contribution of individual steps, such as valuable steps obscured within failed trajectories. To uncover latent information and enable more faithful step-level credit assignment, we propose Graph-based Group Policy Optimization (GraphGPO), which first aggregates all rollout trajectories into a unified state-transition graph and then estimates the distance from each state to the task goal using the global information encoded in the graph. Finally, GraphGPO assigns credit to each edge by estimating a graph-based advantage, based on how much the transition reduces the distance to the task goal. In this way, GraphGPO significantly improves training efficiency and achieves state-of-the-art performance across a range of challenging benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。