arXiv:2606.18810cs.LGcs.AI2026-06被引 2

用自验证轨迹计算梯度权重,让强化学习更精准奖励关键推理步骤。

Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewards

论文配图:Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewards
图 1 · 摘自论文原文
  • 基于模型自身验证轨迹构建自教师,通过KL散度动态调整梯度权重。
  • 在五个基准上比GRPO提升8.1%,比DAPO提升5.9%,泛化能力更强。
  • 无需外部教师或特权信息,适合纯可验证奖励的强化学习场景。

具有可验证奖励的强化学习(RLVR)推动了大语言模型在推理任务中的进展,但代表性方法如GRPO对所有词元分配均匀信用,浪费梯度于常规步骤,低估关键推理环节。现有词元级信用分配方法依赖模型外资源:GRPO变体需过程奖励模型或真实答案;知识蒸馏依赖外部教师(有监督蒸馏)或特权信息(自蒸馏)。这些依赖限制了其在纯RLVR设置下的应用。我们观察到,将模型条件化于自身验证轨迹后,原分布与条件分布间的词元级KL散度可度量,且证明从验证轨迹构建的自教师蒸馏会导致多重验证轨迹下不可行的加权平均解。为此提出SC-GRPO(自条件GRPO),以该KL散度作为乘性权重作用于GRPO梯度。在涵盖数学、代码和代理任务的五个基准上,SC-GRPO持续优于GRPO 8.1%、DAPO 5.9%,且具备更强的分布外性能。此外,其表现优于OPD。

原文摘要 · Abstract (English)

Reinforcement learning with verifiable rewards (RLVR) has driven substantial progress in training LLMs for reasoning tasks, but representative methods such as GRPO assign uniform credit across all tokens, wasting gradient on routine tokens while under-crediting pivotal reasoning steps. Existing token-level credit assignment methods require resources beyond the model's own rollouts. GRPO variants rely on process reward models or ground-truth answers. Knowledge distillation assigns credit through per-token divergence but requires external teachers (On-Policy Distillation) or privileged information (On-Policy Self Distillation). However, these dependencies limit applicability in the pure RLVR setting. We observe that conditioning the model on its own verified trajectories induces a measurable per-token KL divergence between the original and conditioned distributions, and prove that distilling from a self-teacher constructed by verified trajectories leads to infeasible weighted-average solutions when multiple verified trajectories exist. We propose SC-GRPO (Self-Conditioned GRPO), which uses KL divergence mentioned before as a multiplicative weight on GRPO gradients. Across five benchmarks spanning math, code, and agentic tasks, SC-GRPO consistently outperforms 8.1% over GRPO and 5.9% over DAPO with stronger OOD performance. Moreover, SC-GRPO achieves higher performance than OPD.

强化学习大模型推理信用分配可验证奖励

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。