通过两级路由专家机制,缓解多领域持续学习中的遗忘问题。
Separation and Collaboration: Two-Level Routing Grouped Mixture-of-Experts for Multi-Domain Continual Learning
- 两级路由动态分配专家组,防止灾难性遗忘。
- 结合任务标识与原型距离,增强跨任务协作能力。
- 利用多模态大模型识别任务,适合持续学习场景。
多领域持续学习(MDCL)从具有类别集和分布变化的顺序任务中持续获取知识。尽管参数高效微调(PEFT)方法可应对这种双重异质性,但仍面临灾难性遗忘和前向遗忘问题。为此,我们提出两级路由分组专家(TRGE)方法。首先,TRGE动态扩展预训练的CLIP模型,为每个任务分配特定专家组以缓解灾难性遗忘;随着专家数量持续增加,该方法在组内保持专家数量静态,并引入组内路由器以缓解因路由复杂度上升导致的过拟合。同时,设计基于任务标识符和任务原型距离的组间路由策略,动态选择相关专家组并融合输出,增强跨任务协作。其次,为准确获取任务标识符,利用具备强大多模态理解能力的多模态大语言模型(MLLM)生成语义任务描述并识别正确任务标识。最后,为缓解前向遗忘,基于训练进度动态融合未见样本的冻结CLIP模型与TRGE适配器输出,兼顾预训练与学习到的知识。在多种设置下的大量实验表明,该方法在更少可训练参数下优于其他先进方法。
原文摘要 · Abstract (English)
Multi-Domain Continual Learning (MDCL) acquires knowledge from sequential tasks with shifting class sets and distribution. Despite the Parameter-Efficient Fine-Tuning (PEFT) methods can adapt for this dual heterogeneity, they still suffer from catastrophic forgetting and forward forgetting. To address these challenges, we propose a Two-Level Routing Grouped Mixture-of-Experts (TRGE) method. Firstly, TRGE dynamically expands the pre-trained CLIP model, assigning specific expert group for each task to mitigate catastrophic forgetting. With the number of experts continually grows in this process, TRGE maintains the static experts count within the group and introduces the intra-group router to alleviate routing overfitting caused by the increasing routing complexity. Meanwhile, we design an inter-group routing policy based on task identifiers and task prototype distance, which dynamically selects relevant expert groups and combines their outputs to enhance inter-task collaboration. Secondly, to get the correct task identifiers, we leverage Multimodal Large Language Models (MLLMs) which own powerful multimodal comprehension capabilities to generate semantic task descriptions and recognize the correct task identifier. Finally, to mitigate forward forgetting, we dynamically fuse outputs for unseen samples from the frozen CLIP model and TRGE adapter based on training progress, leveraging both pre-trained and learned knowledge. Through extensive experiments across various settings, our method outperforms other advanced methods with fewer trainable parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。