用动态评分表实现长程智能体训练的精准奖励分配
DRACO: Fine-Grained Credit Assignment with Dynamic Rubrics for Long-Horizon Agent Training
- 动态生成评分标准，按轨迹整体评估后分摊到每一步
- 在AppWorld上比基线模型高15.9分，比稀疏奖励训练高5.3分
- 无需验证器或额外训练模块，适合无明确成功信号的复杂任务
强化学习依赖可验证奖励时表现良好，但多数长周期任务缺乏此类信号。本文研究无结果反馈场景，采用多标准评分机制提供奖励。传统方法对每条轨迹仅做一次评分，导致单标量信号在数十步中传播不佳。提出DRACO：通过训练过程中动态生成评分标准，对完成轨迹进行一次评分，并将该评价结果以闭式公式分摊至负责的各步骤，生成差异化步级优势用于GRPO。该分配无需训练归属模块。在AppWorld上，DRACO相较基线模型提升15.9分，比使用稀疏真实奖励的GRPO高5.3分，且自身不使用任何验证器。在跨域的Tau-Bench上，较基线提升5.3分，超越真实奖励训练与其他基于评分的方法。代码已开源。
原文摘要 · Abstract (English)
Reinforcement Learning from Verifiable Rewards works well when a task has a programmatic checker, but most long-horizon agent domains have none. We work in the outcome-blind setting, where ground-truth success signals are not available. Multi-criteria rubrics are a popular way to supply such a reward; they are scored once per trajectory, but a single scalar is a poor signal across tens of steps. We propose DRACO: Distributing Rubric-based Advantage for Credit Optimization. It generates rubrics dynamically during training to track the policy's evolving capability, scores those rubrics once per completed trajectory, and redistributes that judgment over the steps responsible for annotated rubrics to produce differentiated per-step advantages in GRPO. The redistribution is closed-form and does not introduce any trained attribution module. On AppWorld, DRACO gains 15.9 points over the base model and 5.3 points over GRPO trained with a sparse ground-truth reward, despite not using any verifiers itself. On out-of-domain Tau-Bench, it gains 5.3 points over the base model even without a frontier judge, beating both ground-truth-reward training and other rubric-based training settings. The code for DRACO is available at https://github.com/IBM/draco.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。