解决多教师强化学习中能力分配不均问题,提升通用智能体性能。
Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillation
- 通过均衡令牌优化预算,动态分配学习资源。
- 将能力恢复率从35.6%提升至83.4%,显著改善跨领域表现。
- 适合研究多教师强化学习与通用智能体训练的学者使用。
多教师在线策略蒸馏(M-OPD)已成为通过密集的令牌级奖励监督,将领域专精的强化学习专家整合为单一通用学生模型的有前景范式。尽管其实践成功,但多教师能力融合的优化动态仍不清晰,且缺乏严格可复现的方法。本文在SmolLM3-3B-Base上建立受控的M-OPD基准,采用虚拟路由机制,隔离路由模糊性。研究发现显著的能力融合差距:标准M-OPD仅实现相对于领域路由最优集成的35.6%可用潜力,简短任务如指令遵循出现严重退化与过早停滞。关键发现:失败并非源于梯度冲突,而是令牌级优化预算严重错配。该病理性机制由三个独立因素驱动:跨领域的结构序列长度差异、非均匀学习率导致的动态收敛漂移,以及异步策略更新引发的多步奖励陈旧。为解决这些失衡,提出Open-MOPD框架,包含令牌份额平衡、感知差距的动态预算分配和学生奖励刷新机制。三者协同重建跨领域平衡,使头空间恢复率从35.6%提升至83.4%。完整开源端到端后训练方案、训练轨迹与评估套件,可在学术可负担硬件上部署。
原文摘要 · Abstract (English)
Multi-teacher on-policy distillation (M-OPD) has emerged as a promising paradigm for consolidating domain-specialized reinforcement learning (RL) experts into a single generalist student via dense, token-level reward supervision. Despite its practical success, the optimization dynamics governing multi-teacher capability integration remain poorly understood, and open, rigorously reproducible recipes are conspicuously lacking. In this work, we establish a controlled M-OPD benchmark on SmolLM3-3B-Base with oracle routing, isolating capability integration from routing ambiguity. Our investigation reveals a pronounced capability integration gap: standard M-OPD captures only 35.6% of the available headroom relative to a domain-routed oracle ensemble, with concise tasks such as instruction following suffering severe degradation and premature stagnation. Crucially, we show that this failure stems not from gradient conflict, but from a severe misallocation of the token-level optimization budget. This pathology is driven by three orthogonal factors: structural sequence-length disparities across domains, dynamic convergence drift due to non-uniform learning rates, and multi-step reward staleness from asynchronous policy updates. To resolve these imbalances, we introduce Open-MOPD, a principled framework incorporating token-share balancing, gap-aware dynamic budget allocation, and student reward refresh. Together, these mechanisms systematically restore cross-domain balance, elevating headroom recovery from 35.6% to 83.4% in a single deployable student. We fully open-source our end-to-end post-training recipe, training trajectories, and evaluation suites on an academically accessible hardware budget.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。