通过竞争机制提升专家模型路由效率，实现更优训练效果。
CompeteSMoE -- Statistically Guaranteed Mixture of Experts Training via Competition
- 引入竞争机制，让高响应专家优先被选中路由。
- 实验显示在视觉指令微调和语言预训练任务中性能超越现有方法。
- 适合追求高效大模型训练的开发者与研究者使用。
稀疏专家混合（SMoE）为突破模型复杂度瓶颈提供了有效方案，但其训练仍面临路由过程不优的问题：执行计算的专家无法直接影响路由决策。本文提出竞争机制，通过让具有最高神经响应的专家参与竞争来路由令牌。理论上，该机制相比传统Softmax路由具备更优样本效率。我们进一步设计了CompeteSMoE——一种简单高效的训练算法，通过路由器学习竞争策略，在极低训练开销下实现优异性能。在视觉指令微调与语言预训练任务上的广泛实验证明，CompeteSMoE在有效性、鲁棒性与可扩展性方面均优于当前主流SMoE策略。代码已开源：https://github.com/Fsoft-AIC/CompeteSMoE。本工作为之前arXiv:2402.02526版本的改进版。
原文摘要 · Abstract (English)
Sparse mixture of experts (SMoE) offers an appealing solution to scale up the model complexity beyond the mean of increasing the network's depth or width. However, we argue that effective SMoE training remains challenging because of the suboptimal routing process where experts that perform computation do not directly contribute to the routing process. In this work, we propose competition, a novel mechanism to route tokens to experts with the highest neural response. Theoretically, we show that the competition mechanism enjoys a better sample efficiency than the traditional softmax routing. Furthermore, we develop CompeteSMoE, a simple yet effective algorithm to train large language models by deploying a router to learn the competition policy, thus enjoying strong performances at a low training overhead. Our extensive empirical evaluations on both the visual instruction tuning and language pre-training tasks demonstrate the efficacy, robustness, and scalability of CompeteSMoE compared to state-of-the-art SMoE strategies. We have made the implementation available at: https://github.com/Fsoft-AIC/CompeteSMoE. This work is an improved version of the previous study at arXiv:2402.02526
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。