解决大模型跨教师强化学习蒸馏中的风格偏差问题，提升效果稳定性。
Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Models
- 通过滚动层交叉拟合估计风格偏差并减去，实现更精准的蒸馏更新。
- 在AIME 2024和LiveCodeBench v5上分别达到82.4%和63.0%准确率。
- 允许训练与蒸馏使用不同模型，突破教师一致性限制，适合实际部署。
在线策略蒸馏（OPD）提供来自教师模型的密集词元级监督，但其效果依赖于教师一致性——即提供监督的教师需与生成微调数据的教师一致。然而在实践中，微调数据来源混杂或未知时，该条件常被违反。此时即使更强的教师也难以提升性能。我们发现原始教师与参考模型间的差异中，除有用上下文信息外，还存在由表述、格式和推理节奏差异引发的重复性成分。为此提出Lightning OPD 2.0，采用滚动层交叉拟合估计该重复成分，作为风格-词元偏差的操作代理，并在构建词元级蒸馏更新前予以扣除。在数学推理与代码生成基准测试中，Lightning OPD 2.0在跨教师设置下持续优于原版。从Klear-Reasoner-8B-SFT出发，其在AIME 2024上达82.4%，在LiveCodeBench v5上达63.0%。结果表明，Lightning OPD 2.0是可行的跨教师OPD方法，可放宽教师一致性要求，使微调数据生成器与蒸馏教师可独立选择。代码即将开源。
原文摘要 · Abstract (English)
On-policy distillation (OPD) provides dense token-level supervision from a teacher, but its effectiveness can depend on teacher consistency, meaning that the model providing OPD supervision should also have generated the demonstrations used to train the supervised fine-tuning (SFT) reference. However, this condition is frequently violated in practice when SFT data have mixed or unknown provenance or when different models are preferred for SFT data generation and subsequent distillation. In such cross-teacher settings, even a stronger OPD teacher can yield little improvement over the SFT reference. We find that raw teacher--reference disagreement contains potentially useful context-specific teacher evidence as well as a recurring component associated with differences in wording, formatting, and reasoning cadence. We introduce Lightning OPD 2.0 with cross-fitted style residualization, which uses rollout-level cross-fitting to estimate this recurring component as an operational proxy for style-token bias and subtracts it before constructing the token-level OPD update. Across mathematical reasoning and code generation benchmarks, Lightning OPD 2.0 consistently outperforms Lightning OPD in cross-teacher settings. Starting from Klear-Reasoner-8B-SFT, Lightning OPD 2.0 reaches 82.4% on AIME 2024 and 63.0% on LiveCodeBench v5. Together, these results establish Lightning OPD 2.0 as a practical approach to cross-teacher OPD, relaxing teacher consistency as a prerequisite and allowing the SFT data generator and distillation teacher to be selected independently. Code will be released soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。