arXiv:2608.24588cs.LG2026-08

让大模型服务代理更懂每一步操作的贡献,提升多轮任务完成能力。

IAPO: Influence-Aware Policy Optimization for Credit Assignment in Multi-Turn Service Agents

论文配图:IAPO: Influence-Aware Policy Optimization for Credit Assignment in Multi-Turn Service Agents
图 1 · 摘自论文原文
  • 用影响依赖图建模多轮交互中动作与信息流的关系
  • 在三个基准上超越现有强化学习方法,提升任务成功率
  • 适合研究多轮对话系统、智能客服的开发者使用

大型语言模型代理通过与用户和外部工具的多轮交互解决长时序任务。在这些场景中,任务信息常随时间逐步呈现而非初始即完整。服务类代理尤其凸显此挑战:用户可能修正目标,工具响应提供后续决策所需信息。因此,仅凭最终奖励无法判断哪些行为促成了任务完成。现有方法依赖其他轨迹的对比或重采样生成步级信号,但已完成的轨迹已记录信息与错误的传播路径。本文提出影响感知策略优化(IAPO),将每条轨迹建模为可训练动作上的类型化影响依赖图,以用户和工具观测为证据,将支持使用与失败使用结构转化为路由权重,重新分配同一轨迹的层级优势。在 Qwen3-4B 与 Qwen3-8B 上的实验表明,IAPO 在 ${τ^2}$-Bench、UserBench 与 AgentChangeBench 三个服务代理基准上均优于多轮强化学习基线。BFCL-v4 多轮测试进一步显示,性能提升不损害多轮函数调用能力。该工作推进了对多轮交互中信用分配的理解,并提供了从稀疏结果反馈训练服务代理的原理性方法。

原文摘要 · Abstract (English)

Large Language Model (LLM) agents increasingly solve long-horizon tasks through multi-turn interactions with users and external tools. In these settings, relevant task information often unfolds over time rather than being fully specified at the initial prompt. Service agents make this challenge especially concrete: users may clarify or revise their goals, while tool responses provide information needed for subsequent decisions. Thus, a final reward alone cannot indicate which actions contributed to resolving the task. Recent methods rely on comparative evidence from other trajectories or resampled continuations, or on separately constructed step-level learning signals, to refine credit. However, a completed rollout already records how information and errors flow between agent actions. We introduce Influence-Aware Policy Optimization (IAPO), which represents each rollout as a typed influence-dependency graph over trainable agent actions, with user and tool observations serving as evidence. IAPO converts support-use and failed-use structure into routing weights that redistribute the same trajectory-level advantage. Experiments with Qwen3-4B and Qwen3-8B demonstrate superior performance over multi-turn reinforcement learning (RL) baselines across three service-agent benchmarks: ${τ^2}$-Bench, UserBench, and AgentChangeBench. BFCL-v4 Multi-Turn further shows that these gains do not compromise multi-turn function-calling performance. This work advances the understanding of credit assignment in multi-turn user interactions and provides a principled approach to training service agents from sparse outcome feedback.

多轮对话信用分配强化学习大模型代理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。