提出新平衡机制，让相似输入用相同专家，提升训练效率
Load Balancing Mixture of Experts with Similarity Preserving Routers
- 用保留输入关系的损失函数，让相似输入选择相似专家
- 相比主流方法，收敛快36%，专家冗余更低
- 适合大规模稀疏专家模型训练，提升资源利用率
稀疏混合专家（MoE）模型通过仅激活部分参数（专家）实现大模型高效训练。学习型路由器为每个输入分配专家子集，但缺乏辅助平衡机制时，易过度依赖少数专家，严重限制模型容量并降低性能。现有负载均衡方法通常追求每条输入的专家分布接近均匀，但训练中会导致路由行为不一致，使模型浪费容量学习重复知识。本文提出一种新型负载均衡损失，保留输入间的相对关系结构，促使相似输入在训练中保持一致的专家选择。实验表明，将该损失应用于路由器后，收敛速度提升36%，冗余程度显著降低。
原文摘要 · Abstract (English)
Sparse Mixture of Experts (MoE) models offer a scalable and efficient architecture for training large neural networks by activating only a subset of parameters ("experts") for each input. A learned router computes a distribution over these experts, and assigns input tokens to a small subset. However, without auxiliary balancing mechanisms, routers often converge to using only a few experts, severely limiting model capacity and degrading performance. Most current load balancing mechanisms encourage a distribution over experts that resembles a roughly uniform distribution of experts per token. During training, this can result in inconsistent routing behavior, resulting in the model spending its capacity to learn redundant knowledge. We address this by introducing a novel load balancing loss that preserves token-wise relational structure, encouraging consistent expert choices for similar inputs during training. Our experimental results show that applying our loss to the router results in 36% faster convergence and lower redundancy compared to a popular load balancing loss.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。