arXiv:2603.06626cs.LGcs.AI2026-03被引 7

通过预训练路由结构加速MoE模型训练,提升收敛速度与效率。

Grouter: Decoupling Routing from Representation for Accelerated MoE Training

  • 先训练优质路由结构,再固定用于新模型,解耦路由与参数更新。
  • 使预训练数据利用率提升4.28倍,训练吞吐量最高加速33.5%。
  • 适用于大规模MoE模型,尤其适合追求高效训练的工程部署者。

传统Mixture-of-Experts(MoE)训练缺乏结构先验,需同时优化专家权重和搜索最优路由策略,导致收敛慢且不稳定。本文提出Grouter,一种预置路由方法:从完全训练的MoE模型中提炼高质量结构,并作为固定路由器用于目标模型。通过解耦结构优化与权重更新,显著加速模型收敛速度与质量。为增强通用性,引入专家折叠以适配不同模型配置,以及专家调优以平衡不同数据分布下的负载。此外,利用预置路由提供的结构先验,可实施针对性优化,进一步提升训练吞吐量。实验表明,Grouter在性能与效率上均表现优异,使预训练数据利用率提升4.28倍,吞吐量最高加速33.5%,确立预置路由为可扩展MoE训练的基本范式。代码与预训练检查点已公开于https://github.com/JimmyAwoe/Grouter。

原文摘要 · Abstract (English)

Traditional Mixture-of-Experts (MoE) training typically proceeds without any structural priors, effectively requiring the model to simultaneously train expert weights while searching for an optimal routing policy within a vast combinatorial space. This entanglement often leads to sluggish convergence and training instabilities. This paper introduces Grouter, a preemptive routing method that by distilling high-quality structures from fully-trained MoE models and serving as a fixed router for target models. By decoupling structural optimization from weight updates, Grouter significantly accelerates both the speed and quality of model convergence. To ensure the framework's versatility, we also introduce expert folding to adapt Grouter across varying model configurations and expert tuning to rebalance workloads across different data distributions. Furthermore, by leveraging the structural priors provided by preemptive routing, we can implement targeted optimizations to further enhance training throughput. Experiments demonstrate that Grouter achieves superior performance and efficiency which boosts pre-training data utilization by 4.28x and achieves up to 33.5% throughput acceleration, establishing preemptive routing as a fundamental paradigm for scalable MoE training. We publicly release our code and pretrained Grouter checkpoints at https://github.com/JimmyAwoe/Grouter.

MoE路由优化训练加速结构先验

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。