通过异步生成与筛选,让模型蒸馏快8倍还更准。
Near-Policy: Accelerating On-Policy Distillation via Asynchronous Generation and Selective Packing
- 学生模型异步生成数据,训练时用序列打包提升效率
- 引入噪声过滤机制,使训练更稳定,性能比监督微调高8.09%
- 适合追求高效蒸馏且想缩小强化学习探索空间的研究者
自回归模型的标准知识蒸馏常因分布不匹配而效果差。现有在线策略方法虽能缓解此问题,但依赖计算开销大的强化学习框架。为此,本文提出近策略蒸馏(NPD),将学生模型生成与训练解耦,实现基于序列打包的监督微调。然而异步更新会引入策略滞后和样本噪声,导致行为偏离近策略。为在不牺牲效率的前提下解决该问题,NPD结合稀疏学生更新与Δ-IFD过滤机制——一种经验性样本筛选策略,可有效剔除分布外极端样本,防止噪声主导梯度,确保更新处于安全邻近学习区域。实验表明,NPD相比在线策略基线提速8.1倍,性能超越监督微调8.09%。关键在于,该方法显著缩小后续强化学习的探索空间,使openPangu-Embedded-1B达到68.73的领先得分,优于更大规模的Qwen3-1.7B。代码即将开源。
原文摘要 · Abstract (English)
Standard knowledge distillation for autoregressive models often suffers from distribution mismatch. While on-policy methods mitigate this by leveraging student-generated outputs, they rely on computationally expensive Reinforcement Learning (RL) frameworks. To improve efficiency, we propose Near-Policy Distillation (NPD), an asynchronous approach that decouples student generation from training. This reformulation enables Supervised Fine-Tuning (SFT) with sequence packing. However, asynchronous updates inevitably introduce policy lag and sample noise, which can cause the behavior to drift from near-policy toward off-policy. To counteract this without sacrificing efficiency, NPD integrates sparse student updates and the $Δ$-IFD filtering mechanism, a heuristic sample selection mechanism that empirically stabilizes the optimization trajectory. By filtering extreme out-of-distribution samples, $Δ$-IFD prevents noise from dominating the gradients, ensuring updates remain within a safe proximal learning zone. Empirically, the NPD framework achieves a 8.1x speedup over on-policy baselines and outperforms SFT by 8.09%. Crucially, by effectively narrowing the exploration space for subsequent RL, our method enables openPangu-Embedded-1B to reach a state-of-the-art score of 68.73%, outperforming the substantially larger Qwen3-1.7B. Codes will be released soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。