提出谱分解方法，让专家分工更明确，提升大模型效率。
SD-MoE: Spectral Decomposition for Effective Expert Specialization
- 从参数与梯度的谱空间分析问题，发现专家间相似性源于共用主导方向。
- 新方法在多个模型上实现有效专家分工，性能显著提升。
- 无需额外计算开销，可无缝接入Qwen、DeepSeek等主流架构。
Mixture-of-Experts（MoE）架构通过条件计算实现大语言模型的扩展，但实践中专家分工常失效：部分专家功能趋同，另一些则沦为事实上的共享专家，限制了模型有效容量与性能。本文从参数与梯度的谱空间角度分析发现：(1) 专家参数中存在高度重叠的主导谱成分；(2) 梯度子空间在专家间强烈对齐，源于人类语料中普遍存在的低秩结构；(3) 门控机制偏好沿这些主导方向路由输入，进一步抑制分工。为此，我们提出谱解耦式MoE（SD-MoE），在谱空间中分解参数与梯度。SD-MoE在下游任务中提升性能，实现有效专家分工，仅引入微小额外计算开销，可无缝集成至Qwen、DeepSeek等多种现有MoE架构。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) architectures scale Large Language Models via expert specialization induced by conditional computation. In practice, however, expert specialization often fails: some experts become functionally similar, while others functioning as de facto shared experts, limiting the effective capacity and model performance. In this work, we analysis from a spectral perspective on parameter and gradient spaces, uncover that (1) experts share highly overlapping dominant spectral components in their parameters, (2) dominant gradient subspaces are strongly aligned across experts, driven by ubiquitous low-rank structure in human corpus, and (3) gating mechanisms preferentially route inputs along these dominant directions, further limiting specialization. To address this, we propose Spectral-Decoupled MoE (SD-MoE), which decomposes both parameter and gradient in the spectral space. SD-MoE improves performance across downstream tasks, enables effective expert specialization, incurring minimal additional computation, and can be seamlessly integrated into a wide range of existing MoE architectures, including Qwen and DeepSeek.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。