用自蒸馏提升语言模型强化学习的稳定性和效果
Self-Distilled Policy Gradient
- 让模型用自身生成内容作为监督信号，实现全词汇自蒸馏
- 在多个任务上相比基线提升性能，且训练更稳定
- 适合做强化学习中奖励稀疏场景的文本生成研究者
在策略自蒸馏中，语言模型利用特权上下文来监督自身的生成，为稀疏奖励的强化学习提供了密集的监督信号。该方法可形式化为一个辅助的全词汇学生-教师反向KL散度损失。我们提出SDPG框架，结合组相对验证器优势与归一化标准差、精确的全词汇在线自蒸馏以及参考策略KL正则化。实验表明，SDPG在稳定性与性能上均优于RLVR和自蒸馏基线。代码已开源：https://github.com/lauyikfung/SDPG。
原文摘要 · Abstract (English)
On-policy self-distillation, where a language model conditions on privileged context to supervise its own generations, is a promising source of dense supervision for sparse-reward reinforcement learning. Actually, it can be instantiated as an auxiliary full-vocabulary student-to-teacher reverse Kullback-Leibler divergence loss. We therefore propose SDPG, a self-distilled policy-gradient framework that combines group-relative verifier advantages with normalized standard deviation, exact full-vocabulary on-policy self-distillation, as well as reference-policy KL regularization. Empirically, SDPG improves stability and performance over RLVR and self-distillation baselines. The code is available at https://github.com/lauyikfung/SDPG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。