通过谱正则化解决MoE专家坍缩，提升模型模块化与稳定性。
Spectral Manifold Regularization for Stable and Modular Routing in Deep MoE Architectures
- 引入谱范数与稳定秩双重正则化，约束路由流形结构。
- 深度增加时传统线性门控准确率下降4.72%，SR-MoE干扰仅-0.32%。
- 适合需要持续学习的高容量模块化神经网络场景。
Mixture of Experts (MoE) 架构虽能高效扩展神经网络，但易出现专家坍缩问题，即路由机制趋向少数主导专家，导致模型容量下降并引发适应过程中的灾难性干扰。本文提出谱正则化MoE（SR-MoE），通过对路由流形施加几何约束，增强结构模块性。方法采用双重正则化：谱范数约束保证路由函数的Lipschitz连续性，稳定秩惩罚保持专家选择中高维特征多样性。在多种架构规模与数据复杂度下，基于模块化单次适应任务的评估显示，传统线性门控随深度增加准确率下降高达4.72%（因专家纠缠），而SR-MoE维持结构完整性（平均干扰-0.32%）。谱约束促进正向知识迁移，支持局部专家更新而不引起全局性能衰减。SR-MoE为构建高容量、模块化且稳定持续学习的神经网络提供通用方案。
原文摘要 · Abstract (English)
Mixture of Experts (MoE) architectures enable efficient scaling of neural networks but suffer from expert collapse, where routing converges to a few dominant experts. This reduces model capacity and causes catastrophic interference during adaptation. We propose the Spectrally-Regularized Mixture of Experts (SR-MoE), which imposes geometric constraints on the routing manifold to enforce structural modularity. Our method uses dual regularization: spectral norm constraints bound routing function Lipschitz continuity, while stable rank penalties preserve high-dimensional feature diversity in expert selection. We evaluate SR-MoE across architectural scales and dataset complexities using modular one-shot adaptation tasks. Results show that traditional linear gating fails with increasing depth (accuracy drops up to 4.72% due to expert entanglement), while SR-MoE maintains structural integrity (mean interference -0.32%). Our spectral constraints facilitate positive knowledge transfer, enabling localized expert updates without global performance decay. SR-MoE provides a general solution for building high-capacity, modular networks capable of stable lifelong learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。