arXiv:2608.07118cs.AI2026-08

提出新方法精准分配多轮智能体强化学习的信用,提升长任务性能。

How Much, Then Where: Credit-Conserving Action-to-Token Allocation for Multi-Turn Agent Reinforcement Learning

论文配图:How Much, Then Where: Credit-Conserving Action-to-Token Allocation for Multi-Turn Agent Reinforcement Learning
图 1 · 摘自论文原文
  • 分两步分配信用:先定动作整体贡献,再按实际生成内容分配
  • 在三个环境中均超越基线,长序列任务提升最显著
  • 方法稳定易迁移,适配不同模型架构且无需调参

多轮智能体强化学习中的信用分配涉及两个层次:为动作分配轨迹级信用,并将每个动作的信用分配到其对应词元。本文提出 FACTOR,将这两个决策分离。FACTOR 使用检查点校准的时序差分残差,为每个动作分配可回溯至轨迹优势的信用;通过反馈条件下的教师-学生似然差距,将每项信用分配给实际生成的动作词元。动作级归一化保持动作平均系数,防止词元级符号翻转。结合动作均值压缩,消除动作标量代理权重对词元长度的隐式依赖。在行为策略中剪裁前,每个动作的内部动作均值代理等于其 TD 信用。FACTOR 在 ALFWorld、WebShop 和 ScienceWorld 环境中持续优于竞争基线,所有环境-种子组合均支持 FACTOR,最长视野环境下收益最大。相同超参数无需重调即可迁移到更大主干网络和不同模型族。消融实验表明 TD 动作信用是改进主要驱动力,事后词元分配带来互补增益。

原文摘要 · Abstract (English)

Credit assignment in multi-turn agent reinforcement learning operates at two levels: assigning trajectory-level credit to actions and distributing each action's credit across its tokens. In this paper, we introduce FACTOR, which separates these decisions. FACTOR uses checkpoint-calibrated TD residuals to assign per-action credits that telescope to the trajectory advantage, and feedback-conditioned teacher-student likelihood gaps to allocate each credit across the realized action tokens. Per-action normalization preserves the action-average coefficient and prevents token-level sign flips. We pair this construction with an action-mean reduction, removing the implicit dependence of an action's scalar surrogate weight on its token length. At the behavior policy and before clipping, each action's inner action-mean surrogate equals its TD credit. FACTOR consistently improves over competitive baselines across ALFWorld, WebShop, and ScienceWorld, with every environment-seed comparison favoring FACTOR and the largest gains emerging on the longest-horizon environment. The same hyperparameters transfer without retuning to a larger backbone and to a different model family. Ablations identify TD action credit as the dominant driver of the improvement, with hindsight token allocation contributing complementary gains.

强化学习信用分配多轮任务智能体

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。