提高专家粒度能显著增强模型表达能力，理论与实证均支持此结论。
The power of fine-grained experts: Granularity boosts expressivity in Mixture of Experts
- 通过增加每层激活的专家数量（粒度），提升模型表达能力
- 理论证明粒度越高，网络表达能力呈指数级提升
- 适用于追求高效高表达力模型的架构设计者
Mixture-of-Experts（MoE）层在前沿模型架构中日益重要。通过选择性激活参数，它们在保持总参数量增长的同时降低计算成本。本文研究了活跃专家数量（即粒度）的影响，对比了拥有较多专家（如 DeepSeek 中每层 8 个）与较少专家（如 Llama-4 模型中每层 1 个）的架构。我们证明了该设计参数导致网络表达能力存在指数级差异，表明更高粒度的模型更具优势。实验结果验证了理论发现，并展示了这一差距的实际体现。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) layers are increasingly central to frontier model architectures. By selectively activating parameters, they reduce computational cost while scaling total parameter count. This paper investigates the impact of the number of active experts, termed granularity, comparing architectures with many (e.g., 8 per layer in DeepSeek) to those with fewer (e.g., 1 per layer in Llama-4 models). We prove an exponential separation in network expressivity based on this design parameter, suggesting that models benefit from higher granularity. Experimental results corroborate our theoretical findings and illustrate this separation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。