提出简单高效的策略，让模型先学老师思考方式再训练，效果更稳。
Simple-OPD: Demystifying Warm-up for On-policy Distillation
- 用教师生成的思维链初始化学生模型，无需正确答案
- 结合低秩适配与适度训练时长，平衡泛化与适应能力
- 无需改动主流程，可直接插入现有训练中使用
在策略蒸馏（OPD）中，学生模型通过自身生成的数据和教师提供的逐标记监督进行训练，但其效果高度依赖于训练前的预热阶段。本文从数据与训练两个角度揭示预热机制：在数据层面，发现有效预热依赖于教师兼容的思维链监督，即使教师生成的轨迹错误，也能提供与正确轨迹相当的效果，说明预热主要传递的是教师的思维模式而非正确答案；在训练层面，表明使用低秩适配（LoRA）并配合接近饱和的训练周期，相比全参数微调（SFT）能更好平衡领域内适应与域外泛化。基于此，提出Simple-OPD——一种即插即用的预热方法，在正式OPD前，用教师生成的思维链与LoRA对模型进行初始化。跨多种设置的实验验证了该方法的有效性与鲁棒性。
原文摘要 · Abstract (English)
On-policy distillation (OPD) trains a student on its own rollouts with token-level supervision from teacher models, but its effectiveness can depend strongly on the warm-up stage before OPD. In this paper, we demystify warm-up for OPD from both data and training perspectives. For data, we find that effective warm-up relies on teacher-compatible chain-of-thought supervision, and that even incorrect teacher rollouts can provide comparable benefits to correct ones. This suggests that warm-up primarily transfers a teacher-compatible thinking pattern rather than merely correct answers. For training, we show that low-rank adaptation (LoRA) with a near-saturation training duration better balances in-domain adaptation and out-of-distribution generalization than full-parameter SFT. Based on these findings, we propose Simple-OPD, a plug-and-play initialization method that warms up the student on teacher-generated CoT with LoRA before OPD. Experiments across diverse settings demonstrate the effectiveness and robustness of Simple-OPD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。