用稀疏奖励训练老师,密集监督压缩到小模型,提升推理性能
Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Principle for Language-Model Post-Training
- 先用稀疏序列奖励训练强教师模型,再通过密集监督蒸馏到小学生
- 相同学生规模下,新方法在MATH和AIME测试中分别达到79.3%和25.2%
- 适合资源有限时优化标注数据分配,尤其对语言模型微调有指导意义
在标注可验证数据稀缺的场景中,每个标注样本应分配给信息量最大的模型与奖励密度。我们提出一个奖励密度原则:稀疏的序列级奖励适用于能探索并发现更好行为的模型,而密集的词元级教师监督更适合将行为压缩到更小的部署模型。该原则导出简单规则:上游用最强教师模型接收稀疏标注数据,下游以密集监督形式迁移强化行为。我们在Qwen3和Llama模型上通过四阶段流程(教师强化学习、前向KL预热、在线策略蒸馏、可选后桥学生强化学习)验证该规则。固定Qwen3-1.7B学生规模下,经8B教师强化后蒸馏的模型在MATH上达79.3%,高于直接GRPO的75.9%;在AIME 2024上为25.2%,优于19.8%(avg@16)。组件消融显示:替换强化教师损失7.8分,移除前向KL预热损失1.7分,移除在线蒸馏损失3.3分。教师质量排序为:原始教师迁移 < 直接GRPO < 强化教师迁移,该规律在使用Llama-3.3-70B-Instruct作为教师时复现于Llama-3.1-8B-Instruct。核心启示是避免将稀缺标注数据用于准备不足的策略:教师侧用稀疏奖励探索,学生侧用密集监督压缩,学生端仅在桥梁建立后才启用稀疏奖励。
原文摘要 · Abstract (English)
In settings where labeled verifiable training data is the binding constraint, each checked example should be allocated to the model and reward density where it is most informative. We identify a reward-density principle that governs this allocation: sparse sequence-level reward is most useful on models that can explore and discover better behavior, while dense token-level teacher supervision is better suited for compressing that behavior into a smaller deployment model. The principle yields a simple allocation rule: use scarce labeled data upstream on the strongest available teacher, then transfer the reward-shaped behavior downstream as dense supervision. We evaluate this rule through a four-stage workflow -- teacher RL, forward-KL warmup, on-policy distillation, optional post-bridge student RL -- on verifiable math with Qwen3 and Llama models. At fixed Qwen3-1.7B deployment-student size, an RL-improved 8B teacher distilled through the dense bridge outperforms direct GRPO on the same student ($79.3\%$ vs.\ $75.9\%$ on MATH; $25.2\%$ vs.\ $19.8\%$ on AIME~2024, avg@16), while transfer from the same teacher \emph{before} RL underperforms. A component ablation confirms that each stage is load-bearing: replacing the RL-improved teacher with a raw teacher costs $7.8$ MATH points, removing the forward-KL warmup costs $1.7$, and removing on-policy distillation costs $3.3$. The teacher-quality ordering -- raw-teacher transfer $<$ direct GRPO $<$ RL-teacher transfer -- replicates on Llama-3.1-8B-Instruct with a Llama-3.3-70B-Instruct teacher. The operational lesson is to avoid spending scarce labeled data on the least prepared policy: use sparse reward for teacher-side discovery, dense transfer for student compression, and student-side sparse reward only after the bridge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。