渐进式扩展专家数量，让大模型训练更高效省时
EMO: Frustratingly Easy Progressive Training of Extendable MoE
- 训练初期少用专家，逐步动态增加，避免资源浪费
- 在大规模实验中性能媲美固定专家结构，节省训练时间与显存
- 适合追求高效训练的大模型研究者和工程团队
稀疏混合专家（MoE）模型可在不增加计算量的前提下扩展模型规模，因为每条输入的浮点运算量仅取决于k个活跃专家，而非全部E个专家。然而，这种不对称性导致实际训练中出现效率悖论：增加专家数量会显著提升内存和通信开销，使训练变得低效。我们指出，问题部分源于当前训练从一开始就分配过多专家，而早期数据未必能充分利用。为此，我们提出EMO，一种简单的渐进式训练框架，将专家容量视为可扩展内存，在训练过程中逐步扩充专家池。EMO通过显式建模扩展规律，推导出各阶段的计算最优令牌预算以实现渐进扩展。实验表明，EMO在大规模实验中性能达到固定专家设置水平，同时提升实际运行效率。该方法提供了一种出人意料地简单却高效的可扩展MoE训练路径，在保留大专家池优势的同时，降低训练时间和GPU成本。
原文摘要 · Abstract (English)
Sparse Mixture-of-Experts (MoE) models offer a powerful way to scale model size without increasing compute, as per-token FLOPs depend only on k active experts rather than the total pool of E experts. Yet, this asymmetry creates an MoE efficiency paradox in practice: adding more experts balloons memory and communication costs, making actual training inefficient. We argue that this bottleneck arises in part because current MoE training allocates too many experts from the beginning, even though early-stage data may not fully utilize such capacity. Motivated by this, we propose EMO, a simple progressive training framework that treats MoE capacity as expandable memory and grows the expert pool over the course of training. EMO explicitly models sparsity in scaling law to derive stage-wise compute-optimal token budgets for progressive expansion. Empirical results show that EMO matches the performance of a fixed-expert setup in large-scale experiments while improving wall-clock efficiency. It offers a surprisingly simple yet effective path to scalable MoE training, preserving the benefits of large expert pools while reducing both training time and GPU cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。