arXiv:2608.17499cs.AI2026-08

让智能体更懂用户下一步反应,提升多轮交互效果

Towards Better Agents for Multi-Turn User Interaction: The Next User Turn Is More Than Context

论文配图:Towards Better Agents for Multi-Turn User Interaction: The Next User Turn Is More Than Context
图 1 · 摘自论文原文
  • 根据用户下一轮反应提供局部反馈,动态分配奖励
  • 8B和14B模型在九个领域平均提升5.91%和10.22%
  • 特别适合电信等复杂交互场景,零样本迁移有效

面向用户的工具型智能体需在多轮对话中协调对话与工具调用。现有交互强化学习常将每轮轨迹简化为最终奖励,对有效引导、错误及后续修复一视同仁。我们提出反馈感知信用分配(FACA),将每次响应与前序用户-用户段对齐,计算局部归一化反应优势,并将其叠加至已验证的终端结果优势,无需额外评价网络或新采样。在模拟器、可见对话、初始化、采样和优化均匹配的条件下,相较于仅基于结果的交互式GRPO基线,FACA在三个独立训练运行中使九域τ-族平均得分分别在8B和14B模型上提升5.91和10.22个百分点。增益集中于电信领域;8B模型下随机化反应极性会消除该领域收益。该顺序在零样本的Pare-Bench和Co-Gym上同样成立。结果表明,用户下一轮反应可作为可行动态信用信号,显著改善多轮交互智能体。

原文摘要 · Abstract (English)

User-facing tool agents must coordinate dialogue and tool use as user goals unfold over multiple turns. Yet interactive reinforcement learning typically reduces each rollout to a terminal reward, assigning the same credit to effective elicitation, errors, and later repair. The next user turn is more than context: it also provides noisy, temporally local evidence about the preceding user-to-user segment. We introduce \textbf{F}eedback-\textbf{A}ware \textbf{C}redit \textbf{A}ssignment (\textsc{FACA}), which aligns each reaction with that segment, derives a locally normalized reaction advantage, and adds it to verified terminal outcome advantage without an extra critic or rollout. Against an outcome-only Interactive GRPO control matched in simulator, visible dialogue, initialization, rollout, and optimization, \textsc{FACA} improves the nine-domain $τ$-family average across three independently trained runs by 5.91 and 10.22 percentage points at 8B and 14B, respectively. Gains concentrate in Telecom; at 8B, randomizing reaction polarity removes the Telecom gain. The same ordering holds zero-shot on Pare-Bench and Co-Gym. These results demonstrate that next-turn user reactions provide actionable local credit for improving multi-turn user-interacting agents.

多轮对话强化学习智能体反馈机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。