一套可跨架构迁移超参数的框架,让密集模型调好参数后能直接用在专家混合模型上。
Complete-muE: Optimal Hyperparameter Transfer and Scaling for MoE Models
- 通过双桥结构,实现从密集前馈层到专家混合模型的超参数迁移。
- 实验表明,在不同模型规模下超参数最优值变化极小,仅轻微漂移。
- 实用性强:只需在密集模型上调参一次,即可适配各类专家混合模型。
我们提出 Complete-muE,一个面向 Transformer 块中密集前馈网络(FFN)与任意混合专家(MoE)配置间超参数迁移的框架。现有工具如 $μ$P(需固定架构)或 SDE(需固定每步令牌数)无法直接解决 MoE 场景下的超参数迁移问题,因为从密集到 MoE 的转换或总专家数缩放同时改变了架构和每专家令牌数。Complete-muE 采用双桥系统:桥 I 通过归一化路由尺度的主动宽度 $μ$P 实现密集 FFN 与密集 MoE 的映射;桥 II 通过激活专家缩放实现密集 MoE 与稀疏 MoE 的映射,其中一阶 SDE 学习率/权重衰减修正项抵消,仅剩有界残差 $σ_0$ 偏移。由此得到的迁移规则——Complete muE——覆盖了激活专家数、总容量、粒度及共享/分组平衡混合型 MoE 模型的变化,以及通用 Transformer 模型的网络宽度/深度、批量大小和训练时长变化。大量语言模型与扩散模型预训练实验验证,Complete-muE 在不同模型架构与参数量下均保持相对稳定的超参数最优值,仅存在与桥 II 非严格 SDE 行为一致的小幅漂移。实践中,仅需在单一密集参考模型上调参一次,即可近乎最优地迁移至所有 MoE 配置,实现‘调一次密,通用于所有’。这使 MoE 模型在扩大模型容量时,无需昂贵的超参数搜索即可实现比密集模型更快的收敛加速。
原文摘要 · Abstract (English)
We propose Complete-muE, a framework which targets hyperparameter transfer across dense FFN and any Mixture-of-Experts (MoE) setups in transformer blocks. Existing tools such as $μ$P (requires fixed architectue) or SDE (requires fixed per-step token count) cannot directly solve the hyperparameter transfer problem in MoE setups because Dense to MoE transfer or MoE total experts scaling changes both architecture and tokens per expert. Complete-muE solves this challenge with a two-bridge system: Bridge~I maps between dense FFN and Dense MoE by active-width $μ$P with a normalized router scale. Bridge~II maps between Dense MoE and sparse MoE by activated-expert scaling, where the first-order SDE LR/WD correction cancels while a bounded residual $σ_0$ shift remains. The resulting transfer rule, which we term as Complete muE, covers changes in activated experts, total capacity, granularity, and shared/group-balanced hybrids for MoE models as well as network width/depth, batch size, and duration changes for general Transformer models. Extensive language model and diffusion model pretraining experiments confirm that complete-muE yields relatively stable hyperparameter optima across model architectures and parameter counts -- with only minor drift consistent with the non-strict SDE behavior of Bridge~II. In practice this drift is small enough that hyperparameters tuned on a single dense reference transfer near-optimally to all MoE configurations -- \emph{tune dense once, transfer to all} is the practical recipe at the core of Complete-muE. This enables MoE models to achieve accelerated convergence speedup over dense models when scaling model capacity without costly hyperparameter search.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。