提出优化专家模型的内存与推理约束设计原则
Towards Principled Design of Mixture-of-Experts Language Models under Memory and Inference Constraints
- 以总参数和专家稀疏度为核心设计指标
- 专家总数越大，性能越受影响，因核心模型变小
- 应最大化总参数量，同时最小化专家数和稀疏度
现代混合专家（MoE）语言模型的设计主要基于总参数量（内存占用）和活跃参数量（推理成本）。然而，我们发现仅靠这两个因素不足以描述最优架构。通过系统研究，我们证明MoE性能主要由总参数量（$N_{total}$）和专家稀疏度（$s:=n_{exp}/n_{topk}$）决定。此外，$n_{exp}$ 和 $n_{topk}$ 并非在稀疏度中相互抵消；相反，专家总数较大时会略微损害性能，因为为满足内存限制，核心模型的深度和宽度被迫减小。这促使我们提出一个简单的设计原则：在给定约束下，最大化 $N_{total}$，同时最小化 $s$（即最大化 $n_{topk}$）和 $n_{exp}$。我们的发现为解决架构模糊性提供了稳健框架，并指导了MoE设计。
原文摘要 · Abstract (English)
Modern Mixture-of-Experts (MoE) language models are designed based on total parameters (memory footprint) and active parameters (inference cost). However, we find these two factors alone are insufficient to describe an optimal architecture. Through a systematic study, we demonstrate that MoE performance is primarily determined by total parameters ($N_{total}$) and expert sparsity ($s:=n_{exp}/n_{topk}$). Moreover, $n_{exp}$ and $n_{topk}$ do not "cancel out" within the sparsity ratio; instead, a larger total number of experts slightly penalizes performance by forcing a reduction in core model dimensions (depth and width) to meet memory constraints. This motivates a simple principle for MoE design which maximizes $N_{total}$ while minimizing $s$ (maximizing $n_{topk}$) and $n_{exp}$ under the given constraints. Our findings provide a robust framework for resolving architectural ambiguity and guiding MoE design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。