无需验证器,用自蒸馏让大模型高效满足评分标准
Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifiers
- 用带评分标准的教师模型指导无条件学生模型逐词学习
- 仅需每提示一次推理,就能达到与验证器训练相当效果
- 适合验证器成本高或不可靠时,替代传统强化学习训练
评分标准在缺乏唯一正确答案的开放领域中正成为强化学习人类反馈(RLHF)的替代方案。现有基于评分标准的方法依赖大语言模型验证器对每个生成轨迹打分,带来显著训练开销,引入验证器偏差,并将反馈压缩为稀疏的终局信号。本文提出评分引导自蒸馏(RGSD),一种无需验证器的训练方法:以条件化的基础策略作为教师,指导无条件的学生模型进行逐词学习。该方法将稀疏的轨迹级奖励转化为密集的词级学习信号,完全移除训练时的验证器调用。在医学与科学领域的 Qwen-2.5(3B、7B)及 Qwen3-Thinking(4B、8B)模型上,RGSD 在仅使用每提示一次在线推理、无训练期验证器调用的情况下,实现了与基于验证器的 GRPO 相当的评分标准满足度。消融实验表明,原始评分标准提供的教师增强信号强于自生成参考响应;更强的 GRPO 验证器在某些场景下仍可超越 RGSD,表明 RGSD 是在验证器成本或可靠性受限时的互补性替代方案。
原文摘要 · Abstract (English)
Rubrics have emerged as an alternative to RLVR in open-ended domains where a single ground-truth final answer is not available. Existing rubric-based training methods rely on an LLM verifier that scores each rollout against rubrics. This introduces substantial training-time overhead, exposes optimization to verifier-specific biases, and reduces rubric feedback to a sparse end-of-trajectory signal. We propose Rubric-Guided Self-Distillation (RGSD), a verifier-free training method in which the base policy, conditioned on the rubric, serves as the teacher for the unconditioned student. RGSD distills the rubric-conditioned teacher distribution into the student token-by-token, replacing sparse trajectory-level rewards with dense per-token learning signals and removing the LLM judge from the training loop entirely. Across Qwen-2.5 (3B, 7B) and Qwen3-Thinking (4B, 8B) models on medical and science domains, RGSD achieves rubric satisfaction comparable to judge-based GRPO while using one on-policy rollout per prompt and no training-time verifier calls. Ablations show that raw rubrics provide a stronger teacher enrichment signal than self-generated reference responses, while a stronger GRPO judge can outperform RGSD in some settings, positioning RGSD as a complementary verifier-free alternative when verifier cost or reliability is the bottleneck.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。