arXiv:2607.11444cs.CL2026-07

通过重排专家池提升领域微调效果,不增加计算成本。

UMoE:Unlocking Every Expert in Domain-Specific Training

论文配图:UMoE:Unlocking Every Expert in Domain-Specific Training
图 1 · 摘自论文原文
  • 先剔除低相关性专家,再用扰动法重建专家池,保持原参数量。
  • 在数学、代码等5个领域12个评测中,平均准确率提升3.4~6.0点。
  • 无需调参,适合作为通用领域微调方案,尤其适合资源受限场景。

Mixture-of-Experts(MoE)模型在不显著增加计算成本的前提下扩展了容量,已成为前沿大语言模型的关键架构。然而,领域特定的后训练沿用了混合领域预训练形成的专家池:大量专家在目标领域表现不佳,且标准监督微调(SFT)无法改变专家构成。我们提出一种简单、预算不变的流程,在微调前将专家池重新对齐至目标领域。具体包括:(1) 剔除领域相关性最低的专家;(2) 通过基于扰动的专家扩展重建至原始规模;(3) 执行标准SFT。所得模型保持原有专家数、参数量和推理开销。使用单一冻结配方且无需每领域超参调优,UMoE在两种MoE架构(Qwen3-30B-A3B 和 Qwen3.5-35B-A3B)、五个领域(数学、代码、科学、工具使用、代理编程)和十二个基准上均优于直接SFT。典型提升包括数学平均准确率提高3.4点,SWE-bench Verified提升6.0点。在内部强数学语料上,直接SFT已超越Qwen3-30B-A3B-Thinking(82.81 vs. 81.06),而UMoE进一步提升至84.17,额外增益1.36点,表明其对更强的SFT基线也具鲁棒性。数据规模实验显示,随着训练数据增长,优势依然存在。分析发现,直接SFT将大量路由计算分配给低相关性专家子集,可事后移除而平均性能下降甚微;UMoE将冗余算力转化为有效领域能力,实现更低训练损失,且在下游评估所有难度层级均获收益。

原文摘要 · Abstract (English)

Mixture-of-Experts (MoE) models scale capacity without proportional compute cost and have become a key architecture for frontier large language models (LLMs). Yet domain-specific post-training inherits an expert pool shaped by mixed-domain pre-training: a substantial subset of experts contributes little on the target domain, and standard supervised fine-tuning (SFT) leaves the composition of this pool unchanged. We propose a simple, budget-preserving pipeline that realigns the expert pool to the target domain before fine-tuning. Given a target domain, we (1) prune the experts with lowest domain-aligned saliency, (2) regrow the expert pool to its original size through perturbation-based expert expansion, and (3) apply standard SFT. The resulting model preserves the original expert count, parameter count, and inference cost. With a single frozen recipe and no per-domain hyperparameter tuning, UMoE consistently improves over direct sft across two MoE architectures (Qwen3-30B-A3B and Qwen3.5-35B-A3B), five domains (math, code, science, tool-use, and agentic coding), and 12 benchmarks. Representative improvements are 3.4 points in math average accuracy, 6.0 points on SWE-bench Verified. On a strong in-house math corpus, direct sft already surpasses Qwen3-30B-A3B-Thinking (82.81 vs.\ 81.06), yet UMoE further raises the average to 84.17, an additional 1.36 points, demonstrating robustness to a substantially stronger SFT regime. Data-scaling experiments further show that the gain persists as training data grows. Analysis reveals that the direct-SFT model allocates substantial routed-expert compute to a low-saliency subset that can be removed post hoc with little average degradation; UMoE turns this redundant capacity into useful domain capacity and achieves lower training loss, with gains spanning all difficulty levels in downstream evaluation.

MoE微调专家系统高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。