OPD加速秘诀是提前预判模型最优更新路径。
Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation

- 通过模块分配与更新方向双重聚焦,实现高效参数优化。
- 训练初期即对齐最终更新方向,提升收敛稳定性。
- 无需额外参数,3倍提速且性能不降,适合大模型后训练。
在策略蒸馏(OPD)中,现有研究多归因于更密集稳定的监督信号,但其参数层面的效率机制仍不明确。本文提出,OPD的效率源于一种‘预见性’:它在训练早期就建立通往最终模型的稳定更新轨迹。这一预见性体现在两方面:在模块分配层面,识别低效区域并集中更新关键推理模块;在更新方向层面,展现更强的低秩集中性,主导子空间在早期即与最终更新子空间高度对齐。基于此,我们提出EffOPD,一种无需额外可训练模块或复杂调参的插件式加速方法,通过自适应选择外推步长沿当前更新方向推进。该方法平均实现3倍训练加速,同时保持与原OPD相当的最终性能。本研究从参数动态视角揭示了OPD效率本质,为设计更高效的大型语言模型后训练方法提供实践指导。
原文摘要 · Abstract (English)
On-policy distillation (OPD) has emerged as an efficient post-training paradigm for large language models. However, existing studies largely attribute this advantage to denser and more stable supervision, while the parameter-level mechanisms underlying OPD's efficiency remain poorly understood. In this work, we argue that OPD's efficiency stems from a form of ``foresight'': it establishes a stable update trajectory toward the final model early in training. This foresight manifests in two aspects. First, at the \textbf{Module-Allocation Level}, OPD identifies regions with low marginal utility and concentrates updates on modules that are more critical to reasoning. Second, at the \textbf{Update-Direction Level}, OPD exhibits stronger low-rank concentration, with its dominant subspaces aligning closely with the final update subspace early in training. Building on these findings, we propose \textbf{EffOPD}, a plug-and-play acceleration method that speeds up OPD by adaptively selecting an extrapolation step size and moving along the current update direction. EffOPD requires no additional trainable modules or complex hyperparameter tuning, and achieves an average training acceleration of $3\times$ while maintaining comparable final performance. Overall, our findings provide a parameter-dynamics perspective for understanding the efficiency of OPD and offer practical insights for designing more efficient post-training methods for large language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。