动态调整教学数据比例,让多教师蒸馏更高效省力。
D$^3$-MOPD: Adaptive Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation

- 用各领域反KL值变化趋势自动调节训练数据配比
- 性能差距缩小至原来的3%,所需采样步数减少3倍
- 适合多领域知识融合且追求训练效率的场景
多教师在线策略蒸馏(MOPD)通过最小化学生在自身轨迹上的各领域反KL散度,将多个领域专家教师的知识整合到一个学生模型中。现有方法通常在训练前固定各领域的数据混合比例,忽略了不同领域收敛速度差异显著的问题:部分领域早期就趋于稳定,而另一些则在整个训练周期内持续提升。固定比例导致快速收敛领域浪费计算资源,慢速收敛领域又训练不足。为此,我们提出D³-MOPD(动态领域调度),一种零开销调度器,利用训练过程中已生成的各领域反KL信号,实现在线自适应调整领域混合比例。该调度器异步运行于训练流程之外,定期跟踪各领域KL轨迹,估算剩余提升空间和当前改善速率,并据此动态调整领域采样比例,不改变核心训练循环。D³-MOPD可自然扩展至任意数量领域,其预期收益随更多领域引入多样化收敛模式而增长。在基于四个领域专家教师蒸馏得到的Qwen3.6-35B-A3B学生模型上,D³-MOPD将平均学生-教师性能差距缩小至97%(原MOPD为63%),在约3倍减少的采样步数下达到相同峰值性能,并在七项基准中的三项上超越了各领域专家教师。
原文摘要 · Abstract (English)
Multi-teacher on-policy distillation (MOPD) distills several domain-expert teachers into a single student by minimizing per-domain reverse-KL divergence on the student's own rollouts. Existing approaches typically fix the per-domain data mixture before training, overlooking the fact that different domains converge at substantially different rates: some plateau early while others continue to improve throughout the training budget. A fixed mixture therefore wastes compute on fast-converging domains and undertrains slower-converging ones. To address this, we propose D$^3$-MOPD (Dynamic Domain ScheDuling for MOPD), a zero-overhead scheduler that repurposes the per-domain reverse-KL signal already produced during training to adapt the domain mixture online. Running asynchronously outside the training process, an off-process watcher periodically tracks each domain's KL trajectory, estimates remaining headroom and current improvement rate, and accordingly adjusts the domain sampling ratios without altering the core training loop. Our D$^3$-MOPD scales naturally to arbitrary numbers of domains, and the expected benefit grows as more domains introduce more diverse convergence patterns for the scheduler to exploit. On a Qwen3.6-35B-A3B student distilled from four domain-expert teachers, D$^3$-MOPD closes 97% of the average student-to-teacher performance gap, compared with 63% for vanilla MOPD, reaches the same peak performance with an approximately 3$\times$ reduction in rollout steps, and surpasses the specialist teachers on three of seven benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。