分离专家选择与加权，提升稀疏MoE模型性能
Beyond Routing: Decoupling Expert Dispatch and Aggregation in Sparse Mixture-of-Experts
- 将专家路由与输出加权解耦，仅优化加权策略
- 在OLMoE上使语言模型损失降低0.1523，跨数据集稳定提升
- 适用于需要高效推理的MoE模型优化，尤其适合部署场景
稀疏混合专家（MoE）路由器通常用相同分数同时完成专家选择和输出加权。我们研究这两项功能是否应耦合。在预训练的OLMoE-1B-7B上，保持选中的前8个专家身份、专家计算结果及总路由质量不变，仅改变组内加权方式。结构化最优基准使全时域交叉熵提升0.0160 ± 0.0039（三种子）。路由器最高分专家为反事实最优节点仅占17.2%，路由器效用与最优性相关性极低（Spearman 0.030）。因此，我们提出固定调度自适应加权（FDAA），一个301K参数的后计算头，在冻结主干、路由器和专家的前提下，直接以语言建模目标优化。在OLMoE上，FDAA使新WikiText-103测试损失下降ΔCE = -0.1523 ± 0.0031（三种子）；混合域训练在WikiText-103、C4和保留的Penn Treebank上均获得稳健增益。在DeepSeek-V2-Lite上复现该审计，使用前6个路由专家+共享专家，最佳节点余量依然显著，路由器Top1仅在12.5%和16.7%的样本中选出最优专家。单种子复现显示，FDAA在锁定的WikiText和PTB上提升明显，而C4无统计差异。结果支持专家选择与承诺之间的跨架构区分。
原文摘要 · Abstract (English)
Sparse Mixture-of-Experts (MoE) routers commonly use the same scores both to select experts and to weight their already-computed outputs. We study whether these two roles, dispatch and aggregation, should be coupled. On pretrained OLMoE-1B-7B, we keep selected Top-8 expert IDs, expert computation, and total selected router mass fixed and change only within-set aggregation. A structured oracle improves full-horizon cross-entropy by 0.0160 +/- 0.0039 across three seeds; the router's top-scored expert is the counterfactual-best vertex only 17.2% of the time, with router-utility Spearman 0.030. We therefore train Fixed-Dispatch Adaptive Aggregation (FDAA), a 301K-parameter post-compute head optimized directly with the language-modeling objective while freezing the backbone, router, and experts. On OLMoE, FDAA improves fresh WikiText-103 test by Delta CE = -0.1523 +/- 0.0031 across three seeds, and mixed-domain training gives robust gains on WikiText-103, C4, and held-out Penn Treebank under frozen confirmatory evaluation. We also replicate the fixed-dispatch audit on DeepSeek-V2-Lite, which uses Top-6 routed experts plus shared experts. Best-vertex headroom remains significant on WikiText and C4, while router Top1 identifies the best selected expert in only 12.5% and 16.7% of audited examples. In a one-seed mixed-domain replication, FDAA improves locked WikiText and PTB, while C4 is statistically neutral. These results support a cross-architecture distinction between expert selection and expert commitment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。