通过正交与方差损失提升MoE专家专业化，显著改善模型性能。
Advancing Expert Specialization for Better MoE
- 引入正交损失让专家处理不同类型的输入token
- 方差损失增强路由决策差异性，专家分工更明确
- 无需修改结构，提升23.79%性能，适合追求高效MoE的开发者
混合专家（MoE）模型通过仅激活部分专家实现大语言模型的高效扩展。然而，我们发现常用的辅助负载均衡损失常导致专家重叠和过度均匀路由，抑制专家专业化并降低后训练性能。为此，提出一种简单有效的方法，引入两个互补目标：(1) 正交性损失，促使专家处理不同类型的标记；(2) 方差损失，鼓励更具区分度的路由决策。梯度层面分析表明，这两个目标与现有辅助损失兼容，有助于优化训练过程。在多种模型架构和多个基准测试上的实验结果表明，该方法显著增强了专家专业化。值得注意的是，相比带有辅助损失的经典MoE基线，性能提升最高达23.79%，同时在下游任务中保持负载均衡，且无需任何结构修改或额外组件。代码将开源以促进社区发展。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models enable efficient scaling of large language models (LLMs) by activating only a subset of experts per input. However, we observe that the commonly used auxiliary load balancing loss often leads to expert overlap and overly uniform routing, which hinders expert specialization and degrades overall performance during post-training. To address this, we propose a simple yet effective solution that introduces two complementary objectives: (1) an orthogonality loss to encourage experts to process distinct types of tokens, and (2) a variance loss to encourage more discriminative routing decisions. Gradient-level analysis demonstrates that these objectives are compatible with the existing auxiliary loss and contribute to optimizing the training process. Experimental results over various model architectures and across multiple benchmarks show that our method significantly enhances expert specialization. Notably, our method improves classic MoE baselines with auxiliary loss by up to 23.79%, while also maintaining load balancing in downstream tasks, without any architectural modifications or additional components. We will release our code to contribute to the community.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。