系统研究专家数量与粒度对MoE模型的影响，发现只需优化这两点即可。
Slicing and Dicing: Configuring Optimal Mixtures of Experts
- 在2000+次实验中全面测试专家数量、维度和路由策略
- 专家参数越多性能越好，即使活跃参数占比仅1/128
- 专家大小只与活跃参数有关，共享专家等设计影响小
Mixture-of-Experts（MoE）架构已成为大语言模型的标准，但其核心设计选择——专家数量、粒度、共享专家、负载均衡、令牌丢弃等——大多仅在狭窄范围内被单独研究。这些选择是否可独立优化仍不明确。本文首次系统性地开展超过2000次预训练实验，涵盖最大达66亿总参数的模型，全面调整专家总数、专家维度、单层内异构专家规模、共享专家大小及负载均衡机制。结果表明，在所有研究的活跃参数尺度下，性能随总MoE参数增加而持续提升，即使在极端活跃专家参数比例128:1时亦然。其次，最优专家大小几乎不受总参数量影响，仅取决于活跃参数量。第三，其他因素如共享专家、异构专家及负载均衡设置对性能的影响相对较小，但无丢弃路由策略始终带来稳定收益。总体而言，结果建议简化配置：重点优化专家数量与粒度，其余设计影响甚微。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) architectures have become standard in large language models, yet many of their core design choices - expert count, granularity, shared experts, load balancing, token dropping - have only been studied one or two at a time over narrow configuration ranges. It remains an open question whether these choices can be optimized independently, without considering interactions. We present the first systematic study of over 2,000 pretraining runs spanning models up to 6.6B total parameters, in which we exhaustively vary total experts, expert dimension, heterogeneous expert sizing within a single layer, shared expert size and load-balancing mechanisms. We find that at every active-parameter scale that we study, performance consistently improves with total MoE parameters even at extreme active expert parameter ratios like 128.Further, the optimal expert size is nearly invariant to total parameter count and depends only on active parameter count. Third, we see that other choices like shared experts, heterogeneous experts and load-balancing settings have small effects relative to expert count and granularity, although dropless routing yields a consistent gain. Overall, our results suggest a simpler recipe: focus on expert count and granularity, other choices have minimal effect on final quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。