arXiv:2510.26270cs.AI2025-10被引 2

为大模型智能体训练设计了基于图结构的双重信用分配方法,提升决策精准度。

Graph-Enhanced Policy Optimization in LLM Agent Training

  • 构建状态转移图,结合拓扑中心性和语义相似性计算任务关键度得分
  • 在ALFWorld、WebShop等任务上成功率提升1.1%至3.8%,7B模型表现最优
  • 适用于需要多步推理的复杂决策场景,尤其适合降低训练波动

交互环境中多步骤大模型智能体代表了长时序决策的重要进展。现有基于群体的强化学习方法对轨迹内每一步及同终端奖励的轨迹赋予相同信用,忽视实际贡献差异。由于采样轨迹构建的状态转移图中不同状态具有不同结构角色,其影响应区分,并在步骤与轨迹层面转化为任务感知信用。为此,我们提出图增强策略优化(GEPO),实现多步大模型智能体训练中的双层结构信用分配。具体地,GEPO通过状态转移图的介数中心性与任务提示的语义相似性融合,生成状态级任务条件关键度评分;基于此评分,轨迹信用经状态自适应折扣重塑,步骤信用则按后继状态的关键度缩放。实验表明,GEPO在ALFWorld上成功率优于最强基线1.1%,在WebShop上提升3.2%,在搜索增强型问答任务上平均提升3.8%(7B规模)。相比平坦群体方法,GEPO显著降低跨种子方差,使梯度信号聚焦于最关键的步骤。

原文摘要 · Abstract (English)

Multi-step LLM agents in interactive environments represent a crucial step toward long-horizon decision-making. To train such agents, group-based reinforcement learning is widely adopted, which reinforces trajectories with higher relative performance within the group. However, in most existing methods, every step within a trajectory and every trajectory with the same terminal reward receive identical credit, regardless of their actual contributions. Since different states play different structural roles in an online state-transition graph built from sampled trajectories, their impacts should be differentiated and converted into task-aware credit at both the step and trajectory levels. We therefore present Graph-Enhanced Policy Optimization (GEPO), a framework for dual-level structural credit assignment in multi-step LLM agent training. Specifically, GEPO derives a state-level Task-Conditioned Criticality score that combines topological betweenness on the state-transition graph with semantic similarity to the task prompt. Based on this score, trajectory-level credit is reshaped through a state-adaptive discount, while step-level credit is scaled by the criticality of its successor state. Experimental results show that GEPO outperforms the strongest baselines by 1.1\% in success rate on ALFWorld, 3.2\% on WebShop, and 3.8\% on average across search-augmented QA tasks at the 7B scale. Compared with flat group-based methods, GEPO reduces across-seed variance and concentrates gradient signals on the most critical steps.

强化学习大模型智能体信用分配图神经网络

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。