提出CRAFT框架,让智能体在自蒸馏强化学习中更精准地分配奖励。
CRAFT: Counterfactual Credit Assignment from Free Sibling Rollouts for Self-Distilled Agentic Reinforcement Learning
- 利用兄弟轨迹计算反事实重要性,实现带符号的逐标记奖励分配
- 通过动态调节蒸馏权重与KL惩罚,提升训练稳定性与效率
- 可在不改变原有代码的前提下逐步启用新机制,便于对比验证
自蒸馏智能体强化学习通过引入分词级蒸馏损失,以同一策略在特权上下文下的输出作为教师。现有方法仅依赖单个标量——教师与学生对数概率差值来调控该损失,存在双重缺陷:一是事后评估,仅考虑实际执行的轨迹而忽略反事实路径;二是符号盲区,无法识别教师偏好动作是否损害整体表现。本文提出CRAFT,包含三个支柱:第一,反事实标记重要性,复用GRPO已采样的G-1个兄弟轨迹,按对数概率差加权,形成近似无偏的组级反事实优势变化估计,实现零额外计算成本的带符号逐标记信用分配;第二,非对称控制器,随门控活动指数移动平均,动态调整蒸馏权重与参考KL权重,彼此反向调节;第三,逐标记极化KL惩罚,在每一步根据信用符号切换为模式聚焦或模式覆盖更新。每个支柱均有独立开关,关闭时损失与梯度与基线在IEEE-754算术下完全一致,确保性能增益源自算法改进而非实现偏差。论文证明了估计器的一致性与方差上界,提供结构与比特精确可复现性保证,并在三个智能体环境、四种模型规模、五种端到端方法及两个基准上评估CRAFT。其中,Adaptive-CRINGE与CRAFT共享第二支柱,用于隔离反事实贡献的影响。
原文摘要 · Abstract (English)
Self-distilled agentic reinforcement learning augments trajectory-level reward with a token-level distillation loss, using as its teacher the same policy conditioned on privileged context. The prevailing recipe gates this loss by a single scalar, the teacher-student log-probability gap. This signal is doubly limited: it is retrospective, scoring only the realised rollout and never the counterfactual ones, and it is sign-blind, never signalling when a teacher-preferred action would have harmed the trajectory. We introduce CRAFT, a three-pillar credit-assignment scheme that addresses both limitations. Pillar 1, Counterfactual Token Importance, reuses the G-1 sibling rollouts that GRPO already samples and importance-weights them by the log-probability gap to form a self-normalised estimate of the group-level counterfactual change in advantage from up-weighting teacher-preferred actions at each step; this yields a signed per-token credit at near-zero extra compute. Pillar 2 is an asymmetric controller that raises the distillation weight as it lowers the reference-KL weight along an exponential moving average of gate activity, and conversely. Pillar 3 polarises the KL penalty token by token, switching between a mode-seeking and a mode-covering update according to the sign of the credit. Each pillar has an independent switch that, when disabled, renders the loss and gradient byte-identical to the baseline in IEEE-754 arithmetic, so any measured gain is attributable to algorithmic change rather than implementation drift. We prove the estimator's consistency and a variance bound, give structural and bit-exact reproducibility guarantees, and evaluate CRAFT across three agentic environments, four model scales, and five end-to-end methods, plus two tabulated prior-work baselines. Among these is Adaptive-CRINGE, a comparator sharing Pillar 2 with CRAFT, isolating the counterfactual contribution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。