让大模型自动发现并整合多个专业专家，提升多领域表现。
Automatic Expert Discovery in LLM Upcycling via Sparse Interpolated Mixture-of-Experts
- 通过稀疏插值机制自动识别多个专业专家
- 在多个基准上达到领先性能，计算开销更低
- 适合需要高效多领域能力的模型优化场景
我们提出稀疏插值混合专家（SIMoE）指令微调方法，一种端到端算法，可将密集预训练的大语言模型（LLM）转化为具备多领域专长的MoE型模型。在指令微调过程中，SIMoE在指定稀疏性约束下自动识别多个专业化专家，每个专家代表原始模型参数的一个结构稀疏子集，对应数据中的特定领域知识。同时，通过路由网络学习输入相关的专家融合策略，利用丰富的跨专家知识实现更优的下游泛化能力，超越现有基线。实验表明，SIMoE在常见指令微调基准上持续取得最先进性能，且相比所有基线保持更优的性能-计算权衡。
原文摘要 · Abstract (English)
We present Sparse Interpolated Mixture-of-Experts (SIMoE) instruction-tuning, an end-to-end algorithm designed to fine-tune a dense pre-trained Large Language Model (LLM) into a MoE-style model that possesses capabilities in multiple specialized domains. During instruction-tuning, SIMoE automatically identifies multiple specialized experts under a specified sparsity constraint, with each expert representing a structurally sparse subset of the seed LLM's parameters that correspond to domain-specific knowledge within the data. SIMoE simultaneously learns an input-dependent expert merging strategy via a router network, leveraging rich cross-expert knowledge for superior downstream generalization that surpasses existing baselines. Empirically, SIMoE consistently achieves state-of-the-art performance on common instruction-tuning benchmarks while maintaining an optimal performance-compute trade-off compared to all baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。