提出离线强化学习方法，训练效率提升4倍且无需持续运行教师模型。
Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation
- 通过预计算教师模型概率并保证一致性，实现离线策略蒸馏
- 在30小时GPU内使Qwen3-8B达到AIME 2024 69.9%得分
- 适用于MoE架构，单卡即可训练大模型，降低研究门槛
在线策略蒸馏（OPD）是大型语言模型后训练的有效方法，但需持续运行教师服务器，带来巨大基础设施开销。本文探究是否可通过预先计算监督微调（SFT）回放中的教师概率并重复使用来实现离线OPD。发现直接离线化会失效，根源在于忽略的‘教师一致性’条件——必须在SFT和OPD中使用同一教师模型。违反该条件会引入梯度偏差，损害性能。基于此，提出Lightning OPD框架，强制教师一致性，彻底消除对实时教师服务器的需求。理论证明，在教师一致性下，Lightning OPD与标准OPD具有相同最优解，梯度偏差有界，并具备隐式正则化效应，防止策略漂移。实验表明，其在数学推理和代码生成任务上性能媲美标准OPD，训练效率提升4.0倍。从SFT初始化的Qwen3-8B-Base出发，仅用30 GPU小时即达AIME 2024 69.9%；进一步扩展至MoE架构，单个8xH100节点训练Qwen3-30B-A3B至AIME 2024 71.0%，显著降低学术研究门槛。代码已开源：https://github.com/jet-ai-projects/Lightning-OPD。
原文摘要 · Abstract (English)
On-policy distillation (OPD) is an effective post-training paradigm for large language models but requires a live teacher server throughout training, resulting in substantial infrastructure overhead. We investigate whether OPD can be performed offline by precomputing teacher log-probabilities once over SFT rollouts and reusing them during training. We find that naively doing so fails to reliably match standard OPD, and trace the root cause to a previously overlooked condition we term teacher consistency, requiring that the same teacher be used for both supervised fine-tuning and OPD. Violating this condition introduces a gradient bias that degrades performance for both offline and online OPD. Building on this insight, we propose Lightning OPD, an offline on-policy distillation framework that enforces teacher consistency and eliminates the need for a live teacher server entirely. We prove that, under teacher consistency, Lightning OPD shares the same optimum as standard OPD, with bounded gradient discrepancy and an implicit regularization effect that helps prevent policy drift. Experiments on math reasoning and code generation show that Lightning OPD achieves comparable performance to standard OPD while delivering 4.0x higher training efficiency. Starting from an SFT-initialized Qwen3-8B-Base model, Lightning OPD reaches 69.9% on AIME 2024 in just 30 GPU hours. Lightning OPD further scales to MoE architectures, training Qwen3-30B-A3B to 71.0% on AIME 2024 on a single 8xH100 node, substantially lowering the barrier for academic research on LLM post-training. Our code is released at https://github.com/jet-ai-projects/Lightning-OPD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。