通过子空间合并专家，大幅压缩MoE模型且保持性能。
Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging
- 用子空间分解共享权重，解决专家专有参数冲突问题。
- 在Mixtral上减少50%专家，仍保留86%原始性能。
- 适合需要轻量化部署的大模型开发者使用。
Mixture of Experts（MoE）大模型因参数量巨大，面临内存、存储和部署难题。尽管现有专家合并方法可提升效率，但受限于专家特异性带来的参数冲突。本文提出Sub-MoE，一种基于子空间专家合并的新型压缩框架。核心思想是联合对拼接的专家权重进行奇异值分解（SVD），通过提取共享的$U$-矩阵降低冲突，并有效合并专家特异的$V$分量。Sub-MoE包含两个创新阶段：(1) 自适应专家聚类，基于专家输出余弦相似度使用K-means聚类功能一致的专家；(2) 子空间专家合并，先进行专家联合分解以获得组内共享$U$-矩阵，再对各$V$-矩阵进行频率感知合并，最终用合并后的$V$重构专家。该方法在共享子空间中对齐并融合专家，可进一步结合专家内压缩实现推理优化。在Mixtral、DeepSeek和Qwen-1.5|3 MoE模型上的实验证明，Sub-MoE显著优于现有剪枝与合并方法。尤其在Mixtral-8x7B零样本任务中，专家减少50%时仍保持86%性能，减少25%时保持96%性能。代码将开源于https://github.com/lliai/MoERazor。
原文摘要 · Abstract (English)
Mixture of Experts (MoE) LLMs face significant obstacles due to their massive parameter scale, which imposes memory, storage, and deployment challenges. Although recent expert merging methods promise greater efficiency by consolidating multiple experts, they are fundamentally hindered by parameter conflicts arising from expert specialization. In this paper, we present Sub-MoE, a novel MoE compression framework via Subspace Expert Merging. Our key insight is to perform joint Singular Value Decomposition (SVD) on concatenated expert weights, reducing conflicting parameters by extracting shared $U$-matrices while enabling effective merging of the expert-specific $V$ components. Specifically, Sub-MoE consists of two innovative phases: (1) Adaptive Expert Clustering, which groups functionally coherent experts via K-means clustering based on cosine similarity of expert outputs; and (2) Subspace Expert Merging, which first enforces Experts Union Decomposition to derive the shared $U$-matrix across experts in the same group, then pursues frequency-based merging for individual $V$-matrices, and finalizes expert reconstruction using the merged $V$-matrix. In this way, we align and fuse experts in a shared subspace, and can be extended with intra-expert compression for further inference optimization. Extensive experiments on Mixtral, DeepSeek, and Qwen-1.5|3 MoE LLMs demonstrate that our Sub-MoE significantly outperforms existing expert pruning and merging methods. Notably, our Sub-MoE maintains 96\%|86\% of original performance with 25\%|50\% expert reduction on Mixtral-8x7B in zero-shot benchmarks. Code will be released at https://github.com/lliai/MoERazor.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。