通过组合权重与数据稀疏性，提升MoE模型计算效率。
Improving MoE Compute Efficiency by Composing Weight and Data Sparsity
- 利用空算专家实现因果约束下的数据稀疏性，避免训练推理不一致。
- 在视觉语言模型上，相同计算量下训练损失更低，下游性能更优。
- 模型自动学习模态感知路由，无需显式区分视觉与文本数据。
Mixture-of-Experts 层通过权重稀疏性实现计算效率：每个输入仅激活部分专家。数据稀疏性指每个专家仅处理部分输入，但传统专家选择路由会破坏自回归模型的因果性，导致训练与推理不一致。本文通过在路由池中引入零计算（空）专家，当令牌路由至空专家时，对应计算资源不被占用。标准负载均衡目标促使模型均匀使用所有专家（真实与空），从而在期望层面实现数据稀疏性，且无因果违规。在视觉语言模型训练中评估，视觉编码器产生大量低信息量令牌，而文本令牌更密集。在匹配预期浮点运算量下，组合权重与数据稀疏性相比仅权重稀疏性，实现了更高的计算效率前沿，训练损失降低，下游性能提升。模型自发学习模态感知分配，对视觉令牌比文本更激进地路由至空专家，无需显式模态路由。
原文摘要 · Abstract (English)
Mixture-of-Experts layers achieve compute efficiency through weight sparsity: each token activates only a subset of experts. Data sparsity, where each expert processes only a subset of tokens, offers a complementary axis. Expert-choice routing implements data sparsity directly but violates causality in autoregressive models, creating train-inference mismatch. We recover data sparsity within causal token-choice MoE by leveraging zero-compute (null) experts within the routing pool. When a token routes to null experts, those slots consume no compute. The standard load balancing objective trains the model to uniformly use all experts (real and null) therefore creating data sparsity in expectation without the causality violations. We evaluate on vision-language model training, where data heterogeneity is pronounced: vision encoders produce many low-information tokens while text tokens are denser. At matched expected FLOPs, composing weight and data sparsity yields a more compute-efficient frontier than weight sparsity alone, with gains in training loss and downstream performance. The model learns implicit modality-aware allocation, routing vision tokens to null experts more aggressively than text, without explicit modality routing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。