发现大模型专家存在领域专精,可零成本定向激活。
Do Domain-specific Experts exist in MoE-based LLMs?
- 通过实证分析10个MoE大模型,验证领域专用专家存在
- 提出DSMoE框架,不增加推理开销即提升性能
- 适合需要高效领域适配的模型部署场景
在大语言模型时代,混合专家(MoE)架构因其计算效率优势成为训练超大规模模型的有效方法。尽管已有大量研究致力于提升MoE中专家的专精度,但其本质特性及系统性解释仍不明确。本文提出根本性问题:MoE大模型中是否存在领域专用专家?通过对10个参数量介于3.8B至120B的先进MoE大模型进行评估,我们提供了实证证据证明领域专用专家的存在。基于此发现,我们提出无需训练的域引导混合专家(DSMoE)框架,该方法推理成本为零,且在四个开源MoE大模型上均超越了经微调的MoE模型和强基线(如监督微调)。实验表明,该方法在目标与非目标领域均表现优异且具备良好泛化能力,无需额外训练或推理开销。代码已公开于https://github.com/giangdip2410/Domain-specific-Experts。
原文摘要 · Abstract (English)
In the era of Large Language Models (LLMs), the Mixture of Experts (MoE) architecture has emerged as an effective approach for training extremely large models with improved computational efficiency. This success builds upon extensive prior research aimed at enhancing expert specialization in MoE-based LLMs. However, the nature of such specializations and how they can be systematically interpreted remain open research challenges. In this work, we investigate this gap by posing a fundamental question: \textit{Do domain-specific experts exist in MoE-based LLMs?} To answer the question, we evaluate ten advanced MoE-based LLMs ranging from 3.8B to 120B parameters and provide empirical evidence for the existence of domain-specific experts. Building on this finding, we propose \textbf{Domain Steering Mixture of Experts (DSMoE)}, a training-free framework that introduces zero additional inference cost and outperforms both well-trained MoE-based LLMs and strong baselines, including Supervised Fine-Tuning (SFT). Experiments on four advanced open-source MoE-based LLMs across both target and non-target domains demonstrate that our method achieves strong performance and robust generalization without increasing inference cost or requiring additional retraining. Our implementation is publicly available at https://github.com/giangdip2410/Domain-specific-Experts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。