通过动态权重提升大模型推理训练效率,让学习更聚焦中等难度问题。
Restoring the Sweet Spot: Pass-Rate Weighted Self-Distillation for LLM Reasoning

- 用模型自身预测方差作为权重,自动调节学习重点。
- 在科学推理和工具使用任务上,性能提升3.2%~4.3%。
- 无需额外计算,自动生成适应模型能力的训练梯度。
自蒸馏策略优化(SDPO)通过模型自身的反馈条件预测提供细粒度的逐标记信用分配。然而,与具有天然难度感知的GRPO不同,SDPO基于KL的奖励缺乏对问题难度的隐式认知。本文从GRPO的优势归一化角度分析该差距,将可学习性框架扩展至归一化奖励,发现归一化吸收了方差项$p(1-p)$,使各题目的学习能力趋于一致,仅剩$\ oot2\of{p(1-p)}$作为梯度缩放因子。据此提出:将每道题的SDPO损失按$[\\ar{p}(1-\ar{p})]^{1/2}$加权,得到SC-SDPO——一种尺度一致的SDPO变体。该权重可通过带批处理自适应归一化的在线采样零成本获得,隐式构建随模型能力动态演化的课程。在科学推理与工具使用基准测试中,SC-SDPO持续优于SDPO,Qwen3-8B上实现均值@16与多数投票@16分别+3.2/+4.3,OLMo-3-7B上为+1.8/+3.0,且全程保持稳定训练动态。
原文摘要 · Abstract (English)
Self-Distillation Policy Optimization (SDPO) provides dense token-level credit assignment for reinforcement learning with large language models by leveraging the model's own feedback-conditioned predictions as a self-teacher. Unlike GRPO, however, whose group-relative advantage naturally concentrates learning on a sweet spot of intermediate-difficulty questions, SDPO's KL-based advantage lacks an implicit notion of difficulty awareness. We analyze this gap through the lens of GRPO's advantage normalization. Extending the learnability framework to normalized rewards, we show that normalization absorbs the variance term $p(1-p)$, equalizing leading-order learnability across questions and leaving $\sqrt{p(1-p)}$ as the sole residual scaling factor in the per-question gradient. This analysis yields a simple prescription: weight each question's SDPO loss by $[\hat{p}(1-\hat{p})]^{1/2}$, resulting in SC-SDPO, a scale-consistent variant of SDPO. The proposed weights are obtained as a zero-cost byproduct of on-policy rollouts with batch-adaptive normalization, inducing an implicit curriculum that dynamically tracks the model's evolving competence. Experiments on scientific reasoning and tool-use benchmarks demonstrate that SC-SDPO consistently improves over SDPO, yielding gains of +3.2/+4.3 (mean@16/maj@16) on Qwen3-8B and +1.8/+3.0 on OLMo-3-7B, while preserving stable training dynamics throughout optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。