提出新型稳定蒸馏方法,显著提升大模型训练效率与生成质量。
PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformation

- 采用有界幂变换构造稳定奖励函数,解决传统方法梯度爆炸问题。
- 在6个数学推理任务上性能超越原始方法6.37点,加速59.2%。
- 适合追求高效高质大模型蒸馏的开发者与研究者使用。
大语言模型的标准在线策略蒸馏(OPD)通过学生采样令牌估计反向KL目标,获得无偏单样本蒙特卡洛估计器,避免全词汇计算。然而我们发现该估计器存在严重训练病态:采样效率低、生成动态不稳定,且性能远逊于精确全词汇OPD。奖励级诊断表明,问题源于对数比率奖励本身无界,导致早期位置梯度方差极大并持续贯穿训练;标准事后缩放因在畸变后才起效而失效。为此,我们提出PowerOPD:一类源自Box-Cox幂变换的天然有界、符号一致奖励,参数为alpha > 0,其对数比率为alpha → 0的退化极限。在六个数学推理基准和四组Qwen3师生模型上,PowerOPD实现平均精度@8/通过率@8分别达+6.37/+5.71(对比原生OPD)、+3.01/+3.54(对比事后稳定)、+2.59/+8.90(对比全词汇OPD),同时将壁钟时间减少59.2%,峰值显存降低23.1%。较大alpha普遍提升准确率,缩短输出长度,并使梯度范数比原生方法小超过3000倍。
原文摘要 · Abstract (English)
Standard on-policy distillation (OPD) for large language models estimates the reverse-KL objective using student-sampled tokens, yielding an unbiased single-sample Monte Carlo estimator that avoids vocabulary-wide computation. However, we show that this estimator suffers from severe training pathologies in practice: sample inefficiency, unstable generation dynamics, and a substantial performance gap compared to exact full-vocabulary OPD. Reward-level diagnosis traces these pathologies to the log-ratio reward, which is unbounded by construction, producing extremely high-variance gradients concentrated at early positions and persisting throughout training; standard post-hoc scaling fail as they operate only after this distortion occurs. To solve this problem, we propose PowerOPD: a family of natively bounded, sign-consistent rewards from the Box-Cox power transformation, parameterized by alpha > 0, of which the log-ratio is the degenerate alpha -> 0 limit. Across six mathematical reasoning benchmarks and four Qwen3 teacher-student pairs, PowerOPD achieves benchmark-averaged Avg@8/Pass@8 gains of up to +6.37/+5.71 over vanilla OPD, +3.01/+3.54 over post-hoc stabilization, and +2.59/+8.90 over full-vocabulary OPD, while reducing wall-clock time by 59.2% and peak GPU memory by 23.1%. Larger alpha generally improves accuracy, consistently shortens responses, and keeps gradient norms more than 3,000x smaller than vanilla OPD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。