让专家模型多领域学习不遗忘，还能更快更省资源。
Dynamic Expert Specialization: Towards Catastrophic Forgetting-Free Multi-Domain MoE Adaptation
- 动态调整路由机制，保留旧知识同时适应新任务
- 多领域训练下遗忘率降低89%，收敛速度提升68%
- 适合需要持续学习多个任务的大型模型应用
Mixture-of-Experts（MoE）模型通过稀疏门控的专家子网络实现巨大容量，但将其适配到多个领域且避免灾难性遗忘仍是开放挑战。现有方法或计算开销过大，或存在跨领域干扰，或需为每个领域单独训练。我们提出DES-MoE，一种面向多领域适配的动态专家专业化框架。DES-MoE通过三项创新解决遗忘问题：(1) 采用基于知识蒸馏的自适应路由，平衡预训练知识保留与任务特定更新；(2) 实时映射专家-领域相关性，隔离领域特异性梯度；(3) 三阶段自适应微调策略，逐步冻结非专业化参数。在六个领域（数学、代码、法律等）上的评估显示，DES-MoE在统一模型上达到单领域微调性能，当领域数从2增至6时，遗忘率相比全微调降低89%，收敛速度比传统方法快68%。本工作确立了动态专家隔离作为可扩展的多任务MoE适配范式。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models offer immense capacity via sparsely gated expert subnetworks, yet adapting them to multiple domains without catastrophic forgetting remains an open challenge. Existing approaches either incur prohibitive computation, suffer cross-domain interference, or require separate runs per domain. We propose DES-MoE, a dynamic expert specialization framework for multi-domain adaptation of Mixture-of-Experts models. DES-MoE addresses catastrophic forgetting through three innovations: (1) an adaptive router balancing pre-trained knowledge retention and task-specific updates via distillation, (2) real-time expert-domain correlation mapping to isolate domain-specific gradients, and (3) a three-phase adaptive fine-tuning schedule that progressively freezes non-specialized parameters. Evaluated on six domains (math, code, law, etc.), DES-MoE matches single-domain ESFT performance while training one unified model, reduces forgetting by 89% compared to full fine-tuning as domains scale from 2 to 6, and achieves 68% faster convergence than conventional methods. Our work establishes dynamic expert isolation as a scalable paradigm for multi-task MoE adaptation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。