arXiv:2602.15521cs.CLcs.LG2026-02

利用GLU激活模式挖掘稠密模型内在的专家结构,无需训练即可高效转换为稀疏MoE。

ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns

  • 基于GLU激活模式识别通用与专用神经元,分层构建共享与路由专家。
  • 在多个基准上性能超越现有方法,推理效率与精度兼顾。
  • 适合希望快速获得高性能MoE模型的研究者与工程师。

混合专家(MoE)通过稀疏专家激活实现模型容量扩展的同时保持计算效率。然而从零开始训练高质量的MoE成本过高。一种有前景的替代方案是将预训练稠密模型转换为稀疏MoE。现有稠密到MoE的方法分为两类:动态结构剪枝,将稠密模型转为中等稀疏度的MoE架构以平衡性能与推理效率;以及降级(downcycling)方法,用预训练稠密模型初始化高度稀疏的MoE架构。但这些方法破坏了稠密模型内部固有的激活模式,导致专家构造不佳。本文认为,门控线性单元(GLU)机制为稠密到MoE转换提供了天然蓝图。我们发现GLU的细粒度神经元激活模式揭示了粗粒度结构,暴露了一个由持续激活的通用神经元和动态激活的专用神经元组成的内在MoE架构。基于此发现,我们提出ExpertWeaver,一个无需训练的框架,根据激活模式划分神经元,构建具有层自适应配置的共享专家与专用路由专家。实验表明,ExpertWeaver在作为无需训练的动态结构剪枝技术或作为更优的MoE初始化策略时,均显著优于现有方法。

原文摘要 · Abstract (English)

Mixture-of-Experts (MoE) effectively scales model capacity while preserving computational efficiency through sparse expert activation. However, training high-quality MoEs from scratch is prohibitively expensive. A promising alternative is to convert pretrained dense models into sparse MoEs. Existing dense-to-MoE methods fall into two categories: \textbf{dynamic structural pruning} that converts dense models into MoE architectures with moderate sparsity to balance performance and inference efficiency, and \textbf{downcycling} approaches that use pretrained dense models to initialize highly sparse MoE architectures. However, existing methods break the intrinsic activation patterns within dense models, leading to suboptimal expert construction. In this work, we argue that the Gated Linear Unit (GLU) mechanism provides a natural blueprint for dense-to-MoE conversion. We show that the fine-grained neural-wise activation patterns of GLU reveal a coarse-grained structure, uncovering an inherent MoE architecture composed of consistently activated universal neurons and dynamically activated specialized neurons. Leveraging this discovery, we introduce ExpertWeaver, a training-free framework that partitions neurons according to their activation patterns and constructs shared experts and specialized routed experts with layer-adaptive configurations. Our experiments demonstrate that ExpertWeaver significantly outperforms existing methods, both as a training-free dynamic structural pruning technique and as a downcycling strategy for superior MoE initialization.

MoEGLU模型转换专家系统

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。