通过梯度冲突驱动专家拓扑剪枝,实现无指令下自动涌现模块化结构。
Mixture-of-Experts with Gradient Conflict-Driven Subspace Topology Pruning for Emergent Modularity
- 在共享参数空间中动态生成专家,用可学习掩码划分逻辑专家
- 梯度冲突惩罚机制使模型自发修剪干扰路径,形成可解释模块结构
- 无需人工标注任务标签,在无指令场景下仍保持语义专一性
Mixture-of-Experts(MoE)架构通过条件计算实现参数高效,但现有设计存在两大缺陷:结构参数隔离导致灾难性遗忘,指令过拟合使无指令场景性能下降。我们提出CDSP-MoE(冲突驱动子空间剪枝MoE),将孤立的专家容器转变为共享物理子空间中的动态专家实例。基于通用权重子空间假设,CDSP-MoE维持一个超完备参数主干,通过可学习拓扑掩码提取逻辑专家。不同于以往利用梯度冲突进行令牌重分配或优化手术的方法,我们将其作为结构监督信号:滞后的梯度博弈惩罚共享流形中的干扰连接,促使拓扑自发剪除冲突路径,演化出可解释的模块结构。实验表明,CDSP-MoE可在无显式指令条件下实现稳健的内容驱动路由,即使在严格盲推理协议下仍保持语义专一性。代码已开源。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) architectures achieve parameter efficiency through conditional computation, yet contemporary designs suffer from two fundamental limitations: structural parameter isolation that causes catastrophic forgetting, and instruction-overfitting that degrades performance in instruction-free scenarios. We propose CDSP-MoE (Conflict-Driven Subspace Pruning MoE), a framework that addresses these issues through a paradigm shift from isolated expert containers to dynamic expert instantiation within a shared physical subspace. Grounded in the Universal Weight Subspace Hypothesis, CDSP-MoE maintains a super-complete parameter backbone where logical experts are carved out via learnable topology masks. Unlike prior work that uses gradient conflict for token reassignment or optimization surgery, we leverage it as a structural supervisory signal: a Lagged Gradient Game penalizes interfering connections in the shared manifold, enabling the topology to spontaneously prune conflicting pathways and evolve interpretable modular structures. Experimental results demonstrate that CDSP-MoE achieves robust content-driven routing without human-defined task labels, maintaining semantic specialization even under strict blind inference protocols where explicit instructions are absent. Code is available at: https://github.com/konodiodaaaaa1/Conflict-Driven-Subspace-Pruning-Mixture-of-Experts
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。