用多教师在线蒸馏，高效整合大模型多种能力
MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training
- 多个领域专用强化学习教师并行训练，学生在自采数据上蒸馏
- 在Qwen3-30B-A3B上性能超越多种基线，几乎保留所有教师能力
- 支持并行开发、无跨域耦合，适合工业级大模型能力集成
现代大语言模型在后训练阶段依赖强化学习提升特定能力，但如何将多种能力融合到单一模型仍具挑战。现有方法如离线微调和混合强化学习或效率低，或性能下降。本文提出多教师在线蒸馏（MOPD），先为各领域独立训练强化学习教师，再让学生在其自身生成的轨迹上蒸馏这些教师。该方法消除暴露偏差，提供密集优化信号。在Qwen3-30B-A3B上，MOPD优于Mix-RL、Cascade RL、Off-Policy Finetune和Param-Merge基线，几乎完整继承各教师的能力。MOPD还支持领域教师的并行独立开发，避免多领域训练中的跨域耦合问题。该方法已应用于MiMo-V2-Flash的后训练，验证其在前沿大模型能力集成中的实际价值。
原文摘要 · Abstract (English)
Modern large language models (LLMs) rely on reinforcement learning during post-training to push specific capabilities, yet integrating multiple capabilities into one model remains hard. Existing methods, such as Off-Policy Finetune and Mix-RL, are either inefficient or lose performance. In this work, we propose Multi-teacher On-Policy Distillation (MOPD), a post-training paradigm for combining the capabilities of multiple domain RL teachers: we first run per-domain specialised RL to obtain a set of domain teachers, then distill these teachers into the student on its own rollouts. This eliminates exposure bias and provides a dense optimization signal. On Qwen3-30B-A3B, MOPD outperforms Mix-RL, Cascade RL, Off-Policy Finetune, and Param-Merge baselines, inheriting nearly all of each teacher's capability. MOPD also enables parallel, independent development of domain teachers, removing the cross-domain coupling typical of multi-domain post-training. MOPD has been deployed in the post-training of MiMo-V2-Flash, an industrial-scale frontier model, demonstrating its practical value for capability integration in frontier-scale LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。