通过分组路由提升专家模型在指令微调中的专业化与泛化能力
Mixture-of-Clustered-Experts: Advancing Expert Specialization and Generalization in Instruction Tuning
- 先按序列特征分组，再在组内选顶级专家，实现分层路由
- 在多个基准上超越强基线，展现更好泛化性能
- 适合需要高效处理多样输入的大型语言模型场景
稀疏的专家混合（MoE）架构通过仅激活部分子模块实现了高可扩展性，且计算成本不随模块增加而线性上升。然而，在指令微调这种输入异质性显著的场景中，如何提升专家的专业化以增强性能和泛化能力仍是挑战。本文提出混合分组专家（MoCE），采用双阶段路由机制：第一阶段基于序列级特征进行专家组路由，第二阶段在组内以令牌级激活前k个专家。该方法能有效根据知识需求对异质输入进行划分，促进专家组的专业化，同时保留令牌级路由的优势。我们在一系列基准上评估了MoCE，结果表明其持续优于强基线，并展现出更强的泛化能力。详细分析进一步验证了其鲁棒性和有效性。
原文摘要 · Abstract (English)
A sparse Mixture-of-Experts (MoE) architecture has emerged as a highly scalable solution by conditionally activating sub-modules without a proportional increase in computational costs. However, improving expert specialization to enhance performance and generalization remains a challenge for MoE, especially in instruction tuning scenarios characterized by significant input heterogeneity. In this work, we propose the Mixture-of-Clustered-Experts (MoCE) to address this limitation through a dual-stage routing mechanism. The first stage in the mechanism performs expert group routing based on sequence-level features, while the second stage activates the top-$k$ experts within the group at the token level. This approach enables the effective partitioning of heterogeneous inputs based on their knowledge requirements, encouraging expert group specialization while maintaining the advantages of token-level routing. We evaluate MoCE across a comprehensive set of benchmarks, demonstrating its consistent superiority over strong baselines and its enhanced generalization capabilities. Detailed analysis further highlights the robustness and effectiveness of MoCE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。