MoE模型在参数和算力完全相同时，性能可超越密集模型。
Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
- 设计最优MoE架构，使激活专家比例处于性能最佳区间。
- 2B/7B规模下训练近250个模型，验证MoE在同等资源下更优。
- 适用于追求高效推理与高精度的AI研发团队。
混合专家（MoE）语言模型在不增加每标记计算量的前提下，显著扩展了模型容量并取得优异性能。然而，在总参数量、训练算力和数据预算完全相同的情况下，MoE能否超越密集架构？这一问题虽具重要实践价值，却未得到充分研究。本文提出新视角与方法框架，系统分析MoE架构，优化设计以最大化性能。结果发现，当激活率处于最优区间时，MoE模型可在相同参数、训练算力与数据资源下优于其密集对应模型。更重要的是，该最优区间在不同模型规模间保持一致。尽管更多数据有助于提升性能，但可通过数据复用解决。我们通过大规模实验验证：共训练近200个20亿参数模型及超过50个70亿参数模型，累计处理50万亿标记。所有模型检查点均已公开。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) language models dramatically expand model capacity and achieve remarkable performance without increasing per-token compute. However, can MoEs surpass dense architectures under strictly equal resource constraints -- that is, when the total parameter count, training compute, and data budget are identical? This question remains under-explored despite its significant practical value and potential. In this paper, we propose a novel perspective and methodological framework to study this question thoroughly. First, we comprehensively investigate the architecture of MoEs and achieve an optimal model design that maximizes the performance. Based on this, we subsequently find that an MoE model with activation rate in an optimal region is able to outperform its dense counterpart under the same total parameter, training compute and data resource. More importantly, this optimal region remains consistent across different model sizes. Although additional amount of data turns out to be a trade-off for enhanced performance, we show that this can be resolved via reusing data. We validate our findings through extensive experiments, training nearly 200 language models at 2B scale and over 50 at 7B scale, cumulatively processing 50 trillion tokens. All model checkpoints are publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。