通过智能选择蒸馏时机与位置,提升强化学习中无效样本的训练效果。
Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance

- 仅在负向无差异样本中选择性蒸馏,按教师置信度加权
- 只对高熵或师生差异大的词元进行蒸馏,避免过度拟合
- 用教师生成正确轨迹补充正梯度,适合改进语言模型强化训练
基于可验证奖励的强化学习(RLVR)已成为大语言模型后训练的标准范式。尽管组相对策略优化(GRPO)被广泛采用,但其面临奖励信号稀疏问题,当组内所有响应获得相同奖励时梯度完全丢失。在线策略蒸馏(OPD)可通过教师模型提供密集的逐标记监督作为补救,但简单结合GRPO与OPD会导致性能下降,原因有三:并非所有样本都受益于蒸馏;过快拟合教师会削弱RL的探索能力;且OPD优势不对称,抑制多数词元。为此,我们提出RSTG(通过自适应教师引导恢复学习信号),在样本层面将OPD限制在零方差负向提示上,并按教师置信度加权;在词元层面仅针对高学生熵或大师生差异的词元进行蒸馏。此外,我们引入教师生成的正确轨迹进行SFT,注入强化学习无法提供的正梯度信号。实验表明,RSTG在数学任务上比朴素的GRPO+OPD提升+4.02%,代码任务上提升+3.05%。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for post-training large language models (LLMs). While Group Relative Policy Optimization (GRPO) is widely adopted, it suffers from sparse reward signals and loses gradients entirely when all responses within a group receive identical rewards. On-policy distillation (OPD) offers a natural remedy by providing dense, token-level supervision from a teacher model. However, naively combining GRPO with OPD leads to degraded performance, due to three underlying causes: not all samples benefit from distillation; fitting too quickly to the teacher undermines the exploratory capacity of RL; and OPD's advantages are asymmetric, suppressing most tokens. To address these challenges, we propose RSTG (Recovering Learning Signals via Adaptive Teacher Guidance), which applies distillation selectively and precisely where it matters most. At the sample level, OPD is restricted to negative zero-variance prompts with each sample weighted by the teacher's confidence score. At the token level, distillation targets only tokens with high student entropy or large teacher-student divergence. We further augment training with SFT on correct trajectories generated by the teacher model, injecting positive gradient signals where RL yields none. Experiments demonstrate that RSTG substantially outperforms naive GRPO+OPD by +4.02% on math and +3.05% on code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。