解决大模型数学推理中无法解题时梯度消失的问题。
HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation
- 用自蒸馏方法在失败题目上生成正确解法并指导训练
- 在OpenMathInstruct-2数据集上提升通过率0.4%-1.7%
- 适合需要稳定提升复杂问题求解能力的研究者
以强化学习训练的大语言模型在数学推理任务中面临根本挑战：对于完全无法解决的“悬崖提示”（cliff prompts），强化学习梯度会彻底消失，导致无法从这些失败模式中学习。本文提出混合蒸馏策略优化（HDPO），通过特权自蒸馏补充标准强化学习，针对所有推演均失败的提示，利用模型自身提供的真实信息生成特权推演，筛选出正确解法，并将教师模型的逐标记分布蒸馏到学生模型。由于师生模型共享相同权重（仅输入不同），可证明其可实现性差距有界，优于跨模型蒸馏。我们证明，在硬阈值极限下，当过滤条件R=1时，该方法能恢复最优的KL正则化强化学习策略。在OpenMathInstruct-2与Qwen2.5-Math-1.5B-Instruct上的实验显示，HDPO持续提升覆盖率指标（pass@4提升0.8%-1.1%，pass@8提升0.4%-1.7%），同时保持贪婪准确率，且蒸馏权重λ可直接调控探索与利用的平衡。
原文摘要 · Abstract (English)
Large language models trained with reinforcement learning (RL) for mathematical reasoning face a fundamental challenge: on problems the model cannot solve at all - "cliff" prompts - the RL gradient vanishes entirely, preventing any learning signal from reaching these failure modes. We introduce Hybrid Distillation Policy Optimization (HDPO), which augments standard RL with privileged self-distillation targeting cliff prompts. On each training step, HDPO identifies prompts where all rollouts fail, generates privileged rollouts by providing the model with ground-truth information, filters for correct solutions, and distills the teacher's token-level distribution into the student. Because teacher and student share the same weights - differing only in their input - the realizability gap is provably bounded, unlike cross-model distillation. We prove that R=1 filtered privileged generation recovers the optimal KL-regularized RL policy in the hard-threshold limit. Experiments on OpenMathInstruct-2 with Qwen2.5-Math-1.5B-Instruct show that HDPO consistently improves coverage metrics (pass@4 by +0.8-1.1%, pass@8 by +0.4-1.7%) while maintaining greedy accuracy, with the distillation weight lambda providing direct control over the exploration-exploitation tradeoff.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。