提出新路由方法，让专家模型更高效且不丢数据。
Maximum Score Routing For Mixture-of-Experts
- 将路由问题建模为最小成本最大流，用软Topk实现动态分配。
- 在相同计算量下，训练损失更低，评估得分更高。
- 适合追求高效率和负载均衡的大型专家模型研究者。
稀疏激活的混合专家（MoE）模型通过可微分的稀疏变换动态将输入令牌分配给前k个专家，可在保持计算效率的同时扩展模型容量。传统MoE模型施加专家容量限制以确保GPU友好计算，但会导致容量饱和时丢弃令牌，并因专家使用不足而产生填充，降低硬件效率。取消容量限制虽能缓解丢包，却损害负载均衡与计算效率。为此，我们提出最大得分路由（MaxScore），将路由建模为最小成本最大流问题，并引入SoftTopk算子。MaxScore克服了迭代重路由与最优传输方法的根本局限，在等效浮点运算量下实现更低的训练损失和更高的评估分数，优于有无容量约束的基线模型。实现细节与实验配置详见https://github.com/dongbw18/MaxScore.git。
原文摘要 · Abstract (English)
Routing networks in sparsely activated mixture-of-experts (MoE) dynamically allocate input tokens to top-k experts through differentiable sparse transformations, enabling scalable model capacity while preserving computational efficiency. Traditional MoE networks impose an expert capacity constraint to ensure GPU-friendly computation. However, this leads to token dropping when capacity is saturated and results in low hardware efficiency due to padding in underutilized experts. Removing the capacity constraint, in turn, compromises load balancing and computational efficiency. To address these issues, we propose Maximum Score Routing ($\mathbf{MaxScore}$), a novel MoE routing paradigm that models routing as a minimum-cost maximum-flow problem and integrates a SoftTopk operator. MaxScore resolves the fundamental limitations of iterative rerouting and optimal transport formulations, achieving lower training losses and higher evaluation scores at equivalent FLOPs compared to both constrained and unconstrained baselines. Implementation details and experimental configurations can be obtained from $\href{https://github.com/dongbw18/MaxScore.git}{MaxScore}$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。