用前层注意力头初始化多个路由，提升MoE模型复用性能
Router Upcycling: Leveraging Mixture-of-Routers in Mixture-of-Experts Upcycling
- 从前置注意力层提取多个路由，协同分配令牌
- 在多个基准上实现当前最优（SOTA）性能
- 适合想低成本提升MoE模型效果的研究者
Mixture-of-Experts (MoE) 模型因其动态资源分配和在多样化任务中的优异表现，在深度学习中受到广泛关注。然而，高效训练这些模型仍具挑战性。为减少训练开销，已提出 MoE upcycling 技术以复用并改进现有模型组件。尽管如此，如线性路由等简单路由在复杂路由任务中表现不佳。为此，我们提出一种新型路由技术 Router Upcycling，以增强 MoE upcycling 模型的性能。该方法在复用过程中，从先前注意力层的注意力头初始化多个路由，这些路由以类注意力方式协同将令牌分配给专用专家。每个令牌被转化为多种查询，并与专家特征（作为键）对齐。实验结果表明，该方法在多个基准上达到当前最优（SOTA）性能，显著优于其他复用基线。
原文摘要 · Abstract (English)
The Mixture-of-Experts (MoE) models have gained significant attention in deep learning due to their dynamic resource allocation and superior performance across diverse tasks. However, efficiently training these models remains challenging. The MoE upcycling technique has been proposed to reuse and improve existing model components, thereby minimizing training overhead. Despite this, simple routers, such as linear routers, often struggle with complex routing tasks within MoE upcycling. In response, we propose a novel routing technique called Router Upcycling to enhance the performance of MoE upcycling models. Our approach initializes multiple routers from the attention heads of preceding attention layers during upcycling. These routers collaboratively assign tokens to specialized experts in an attention-like manner. Each token is processed into diverse queries and aligned with the experts' features (serving as keys). Experimental results demonstrate that our method achieves state-of-the-art (SOTA) performance, outperforming other upcycling baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。