提出计算增益度量,实现高效MoE模型的精准规模预测。
Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models
- 引入效率增益(EL)量化MoE相比稠密模型的计算优势。
- 发现EL受专家激活率与总算力预算支配,符合幂律关系。
- 构建统一缩放定律,指导低资源下高性能MoE模型设计。
Mixture-of-Experts(MoE)通过解耦参数总量与计算成本,成为高效扩展大语言模型的主流架构。然而,这种解耦带来关键挑战:难以预测特定MoE配置(如专家激活率和粒度)下的模型容量。为此,本文提出效率增益(EL),用于量化MoE模型相对于稠密模型的计算优势。我们开展大规模实证研究,训练超过300个模型,最大达280亿参数,系统分析MoE架构配置与EL的关系。结果表明,EL主要由专家激活率和总计算预算决定,二者均遵循可预测的幂律;而专家粒度呈非线性调节作用,存在明确最优区间。基于此,我们建立统一缩放定律,可准确预测给定配置下MoE的EL。为验证该定律,我们设计并训练了仅含0.85亿活跃参数的Ling-mini-beta模型,并与6.1亿参数的稠密模型在相同1万亿高质量令牌数据集上对比。结果表明,前者性能媲美后者,但计算资源消耗减少超7倍,验证了缩放定律的准确性。本工作为高效MoE模型的规模化提供了理论与实证基础。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) has become a dominant architecture for scaling Large Language Models (LLMs) efficiently by decoupling total parameters from computational cost. However, this decoupling creates a critical challenge: predicting the model capacity of a given MoE configurations (e.g., expert activation ratio and granularity) remains an unresolved problem. To address this gap, we introduce Efficiency Leverage (EL), a metric quantifying the computational advantage of an MoE model over a dense equivalent. We conduct a large-scale empirical study, training over 300 models up to 28B parameters, to systematically investigate the relationship between MoE architectural configurations and EL. Our findings reveal that EL is primarily driven by the expert activation ratio and the total compute budget, both following predictable power laws, while expert granularity acts as a non-linear modulator with a clear optimal range. We integrate these discoveries into a unified scaling law that accurately predicts the EL of an MoE architecture based on its configuration. To validate our derived scaling laws, we designed and trained Ling-mini-beta, a pilot model for Ling-2.0 series with only 0.85B active parameters, alongside a 6.1B dense model for comparison. When trained on an identical 1T high-quality token dataset, Ling-mini-beta matched the performance of the 6.1B dense model while consuming over 7x fewer computational resources, thereby confirming the accuracy of our scaling laws. This work provides a principled and empirically-grounded foundation for the scaling of efficient MoE models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。