让教师和学生动态互学，提升大模型推理能力
DualOPSD: Adaptive Privileged Teachers for On-Policy Self-Distillation
- 教师与学生交替更新，动态适应彼此分布变化
- 在多个数学基准上，准确率最高提升23.61点
- 适合需要强推理的大模型训练场景
在非思考模式下，对Qwen3-8B的实验显示，DualOPSD相较于OPSD在AIME 2024、AIME 2025和HMMT 2025上的avg@12分别提升了23.61、13.89和10.00分。1.7B和4B规模的实验表明，性能增益随模型规模增加而增强。所有三个规模下，DualOPSD均减少了截断现象。4B模型的诊断分析显示，师生间双向KL散度均下降。
原文摘要 · Abstract (English)
On-policy self-distillation (OPSD) uses a privileged copy of the student model to provide dense supervision without an external teacher. OPSD keeps this privileged teacher fixed, even though the student distribution and output style change during training. We propose DualOPSD, an asymmetric alternating framework that adapts both policies. The student first learns from the privileged teacher. The teacher then moves toward the updated student distribution on the same student trajectory. This update makes later supervision responsive to the learner and does not require another rollout. On Qwen3-8B in non-thinking mode, DualOPSD improves avg@12 over OPSD by 23.61, 13.89, and 10.00 points on AIME 2024, AIME 2025, and HMMT 2025. Results at 1.7B and 4B show that the accuracy gain depends on model scale. Across all three scales, DualOPSD reduces truncation. The 4B diagnostic also shows lower KL in both directions between the teacher and student.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。