arXiv:2609.03383cs.LG2026-09

TIGPO让大模型长任务智能体持续积累经验,提升决策准确性。

TIGPO: Temporal Instance-Graph Policy Optimization for Long-Horizon LLM Agents

  • 构建跨周期的动态状态转移图,融合历史与当前探索数据
  • 在ALFWorld和WebShop上性能超越现有方法,成功率显著提升
  • 适合需要长期规划的智能体任务,如复杂交互式应用

基于图的策略优化通过将轨迹组织为状态转移图,改善了长时序大模型智能体的信用分配。然而,现有方法在每次策略更新时独立构建图结构,丢弃早期发现的转移信息,且优势估计局限于小批次内的局部轨迹组。本文提出时序实例图策略优化(TIGPO),将图结构优化扩展至多轮策略更新中。TIGPO为每个任务维护一个持久化转移图,使不同时期策略发现的有效转移共同参与当前轨迹的信用分配。为促进当前探索与历史经验的主动关联,TIGPO在固定采样预算下分配探索槽与重访槽:对每个重访任务,将当前轨迹组与对应的早期探索组配对,构建跨时序参照。该扩大的参照组可稳定小批量下的相对优势估计,并直接捕捉训练阶段间的策略改进。历史转移与得分仅作为结构参考,不用于策略损失中的重放。在ALFWorld和WebShop上的实验表明,TIGPO持续优于先前的组级与图级策略优化方法。

原文摘要 · Abstract (English)

Graph-based policy optimization improves credit assignment for long-horizon LLM agents by organizing rollout trajectories into state-transition graphs. However, existing methods construct graphs independently within each policy update, discarding transitions discovered by earlier policies and limiting advantage estimation to small, batch-local rollout groups. We propose \emph{Temporal Instance-Graph Policy Optimization} (TIGPO), which extends graph-based credit assignment across policy updates. TIGPO maintains a persistent transition graph for each task, allowing valid transitions discovered by different policy versions to jointly determine credit for current rollouts. To actively reconnect current exploration with historical experience, TIGPO allocates a fixed rollout budget between Exploration slots for ordinary task sampling and Revisit slots for delayed reattempts of previously explored tasks. For each revisit, TIGPO pairs the current rollout group with its corresponding earlier Exploration group to construct a cross-temporal reference. The enlarged reference is designed to stabilize relative advantage estimation under small rollout groups, while comparison on the same task directly captures policy improvement across training stages. Historical transitions and scores serve only as structural and detached statistical references and are never replayed in the policy loss. Experiments on ALFWorld and WebShop demonstrate that TIGPO consistently outperforms prior group-based and graph-based policy optimization methods.

长程规划策略优化大模型智能体图神经网络

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。