用抽象技能指导强化学习,让模型自己学会高效解题。
Distill Skills into Weights, Not Prompts: Abstract Skills as Privileged Signals for On-Policy Self-Distillation

- 用技能卡片作为显式信号,引导模型在策略内自蒸馏。
- 在数学基准上,性能比GRPO提升最高12.01分,1.7B模型仍领先3.77分。
- 适合需要高精度推理的强化学习任务,尤其在奖励信号弱时有效。
当回溯组全部正确或全部错误时,可验证奖励的强化学习缺乏相对信号,这在实验中占63.0%至68.0%。我们提出SKALD(技能锚定潜在蒸馏)框架,使用Qwen3-Base模型的两种上下文视图:仅含问题的学生与基于抽象、显式答案过滤技能卡的教师。学生在自身前缀上训练,将技能带来的优势转移到共享参数中,测试时无需特权输入。为稳定上下文引起的分布偏差,SKALD采用渐进式指数倾斜目标,降低学生概率极低的教师偏好词权重;随着倾斜消失,收敛至教师交叉熵并恢复前向KL学生梯度。一个经验门控仅在验证回溯估计出正教师优势时激活蒸馏。在五个保留数学基准上,SKALD在0.6B、1.7B、4B模型下分别比GRPO提升+2.46、+4.85、+12.01。在1.7B模型下,仅零方差蒸馏即可恢复84.7%的全收益,而SKALD仍比同等算力的GRPO高+4.06,超出上下文技能暴露+3.77。结果表明,抽象技能在群体相对奖励无信息时提供密集监督。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards yields no group-relative signal when rollout groups are uniformly correct or uniformly wrong, which account for 63.0-68.0% of groups in our experiments. We propose SKALD (Skill-Anchored Latent Distillation), an on-policy self-distillation framework that uses two context views of the same Qwen3-Base model: a question-only student and a teacher conditioned on an abstract, explicit-answer-filtered skill card. The student is trained on its own prefixes, transferring the skill-induced advantage into shared parameters without privileged input at test time. To stabilize context-induced distribution mismatch, SKALD employs an annealed exponentially tilted objective that downweights teacher-preferred tokens with very low student likelihood; as the tilt vanishes, it converges to teacher cross-entropy and recovers the forward-KL student gradient. An empirical gate activates distillation only when verified rollouts estimate a positive teacher advantage. Across five held-out mathematics benchmarks, SKALD improves overall avg@8 over GRPO by +2.46, +4.85, and +12.01 at 0.6B, 1.7B, and 4B, respectively. At 1.7B, zero-variance-only distillation recovers 84.7% of the full gain, while SKALD remains +4.06 above FLOP-matched GRPO and exceeds contextual skill exposure by +3.77. These results show that abstract skills provide dense supervision where group-relative rewards become uninformative.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。