给智能体动作按语义角色打分,让奖励更精准。
TRIAGE: Role-Typed Credit Assignment for Agentic Reinforcement Learning

- 按动作类型分类:关键进展、有用探索、无进展基建、倒退行为
- 在成功轨迹中识别出冗余操作,减少10%~14%的无效交互
- 适合需要精细奖励设计的复杂任务智能体训练
智能体强化学习需为环境交互动作(如搜索、点击、导航)分配信用。标准GRPO仅以最终验证结果作为统一优势信号,但该信号结构不完整:在失败轨迹中惩罚有效探索,在成功轨迹中强化冗余或倒退动作。本文提出TRIAGE,一种基于语义角色的信用分配框架,在结果信用基础上增加角色轴。通过结构化裁判将每个片段分类为关键进展、有用探索、无进展基建或回归,并用固定的角色条件规则映射为有界段级过程奖励。这保持验证结果作为优化方向的同时,修正了仅依赖结果信用的两大盲点。我们证明贝叶斯最优角色可测量校正是段级优势残差在角色变量上的L2投影,且TRIAGE的固定角色常数近似此投影,当裁判可靠时可降低优势估计误差,从而获得更低方差策略梯度。在ALFWorld、Search-QA和WebShop上,TRIAGE对两种策略模型均提升成功率,优于标量裁判奖励和共享主干值函数基线。消融实验表明收益主要来自角色分类而非简单增加密集奖励:可靠检测成功轨迹中的回归是主要贡献,探索信用带来持续次要收益;在完成的ALFWorld和WebShop轨迹中,相比GRPO分别减少10.4%和14.8%的环境交互轮次。
原文摘要 · Abstract (English)
Agentic reinforcement learning requires assigning credit to environment-facing actions such as searches, clicks, edits, navigation commands, and object interactions. Standard GRPO uses the final verifier outcome as a uniform advantage over all action tokens. This outcome signal is useful but structurally incomplete: it punishes useful exploration in failed rollouts and reinforces redundant or regressive actions in successful rollouts. We propose TRIAGE, a role-typed credit assignment framework that adds a semantic role axis to outcome credit. A structured judge classifies each segment as decisive progress, useful exploration, no-progress infrastructure, or regression, and a fixed role-conditioned rule maps these labels to bounded segment-level process rewards. This keeps verifier outcomes as the source of optimization direction while correcting the two main blind spots of outcome-only credit. We further show that the Bayes-optimal role-measurable correction is the L2 projection of the per-segment advantage residual onto the role variable, and that TRIAGE's fixed role constants approximate this projection, reducing advantage estimation error whenever the judge is reliable; we connect this to lower-variance policy gradients. Across ALFWorld, Search-QA, and WebShop, TRIAGE improves success rates over GRPO for two policy models and outperforms both a scalar judge-derived process reward and an outcome-supervised shared-backbone value baseline. Ablations show that the gain comes from role typing rather than merely adding dense rewards: reliable detection of regression inside successful trajectories is the dominant contributor, while exploration credit provides a consistent secondary gain; on completed ALFWorld and WebShop rollouts, TRIAGE also reduces environment-facing turns by an additional $10.4\%$ and $14.8\%$ relative to GRPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。