解决专家模型在增量学习中路由漂移问题,提升旧知识保持与新知识适应能力。
Stable Routing for Mixture-of-Experts in Class-Incremental Learning

- 通过敏感性引导的路由对齐,稳定旧类别路由分布。
- 引入非对称容量正则化,提升新专家池利用率。
- 在4个基准上显著提升平均与最后准确率,适合需要持续学习的场景。
类增量学习(CIL)要求模型在顺序学习新类别时保留已有知识。近年来,结合预训练模型与混合专家(MoE)的方法受到关注:通常通过扩展专家并用路由器分配权重。然而,现有方法常忽略专家扩展带来的路由漂移问题——新专家引入后,路由器可能将旧类别样本重新分配给新专家,破坏原有专家结构,造成干扰,即使旧专家冻结也如此。我们认为,可扩展的MoE在CIL中需具备两个互补特性:稳定旧类别路由以保知识,充分利用容量以适配新类别。为此,我们提出面向可扩展MoE的稳定路由框架StaR-MoE。通过敏感性感知的路由对齐,StaR-MoE利用敏感性引导约束,使当前旧类别路由行为与历史路由分布对齐。同时,引入非对称容量正则化,促进扩展专家池的有效利用而不损害类别特异性路由专一性。在四个标准CIL基准上的大量实验表明,StaR-MoE始终优于当前最优方法,在平均与最后准确率上均有提升,凸显了稳定路由的重要性。
原文摘要 · Abstract (English)
Class-incremental learning (CIL) requires models to learn new classes sequentially while preserving prior knowledge. Recently, approaches that combine pre-trained models with mixture-of-experts (MoE) have received increasing attention in CIL: they typically expand experts during learning and employ a router to assign weights across experts. However, existing MoE methods often overlook routing drift induced by expert expansion. Once new experts are introduced, the router may reassign samples from earlier classes to newly added experts, thereby perturbing previously established expert compositions and causing interference even when old experts remain frozen. We argue that expandable MoE in CIL requires two complementary properties: stable old-class routing for knowledge preservation and sufficient capacity utilization for new-class adaptation. To this end, we propose Stable Routing for MoE (StaR-MoE), a routing-level framework for expandable MoE in CIL. By incorporating sensitivity-aware routing alignment, StaR-MoE aligns current old-class routing behavior with historical routing distributions through sensitivity-guided constraints. Complementarily, StaR-MoE introduces asymmetric capacity regularization to encourage effective utilization of the expanded expert pool without compromising class-specific routing specialization. Extensive experiments across four standard CIL benchmarks demonstrate that StaR-MoE consistently improves both average and last accuracy over state-of-the-art methods, highlighting the importance of stable routing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。