动态专家演化框架缓解视觉语言模型持续学习中的遗忘问题
DIMoE-Adapters: Dynamic Expert Evolution for Continual Learning in Vision-Language Models
- 构建可动态演化的稀疏专家池，提升模型适应新任务能力
- 在多个数据集上相比现有方法准确率提升3-8个百分点
- 适合需要长期更新、避免灾难性遗忘的视觉语言应用
持续学习使视觉语言模型能积累知识并适应不断变化的任务，而无需从头训练。然而，在多领域任务增量学习中，较大的领域差异加剧了稳定与可塑性的矛盾。现有方法多依赖固定架构和静态参数分配，限制了对新领域的适应能力，并加重灾难性遗忘。为此，我们提出DIMoE-Adapters，一种动态增量专家混合适配器框架，引入动态专家演化范式以平衡稳定性与可塑性。该范式通过两个协同组件实现：自校准专家演化（SCEE）和原型引导专家选择（PGES）。SCEE通过专家优化动态构建并演化稀疏专家池，提升可塑性同时减少冗余容量；PGES基于SCEE生成的专家池控制专家使用，增强对已见与未见任务的稳定性。大量实验表明，DIMoE-Adapters在多种设置下均优于先前最先进方法。
原文摘要 · Abstract (English)
Continual learning enables vision-language models to accumulate knowledge and adapt to evolving tasks without retraining from scratch. However, in multi-domain task-incremental learning, large domain shifts intensify the stability-plasticity dilemma. Most existing methods rely on fixed architectures with statically allocated parameters, which limits adaptation to new domains and aggravates catastrophic forgetting. To address these challenges, we propose DIMoE-Adapters, a Dynamic Incremental Mixture-of-Experts Adapters framework that introduces a dynamic expert evolution paradigm to balance stability and plasticity. This paradigm is implemented through two collaborative components: Self-Calibrated Expert Evolution (SCEE) and Prototype-Guided Expert Selection (PGES). SCEE constructs and evolves a sparse expert pool through expert optimization dynamics, improving plasticity while reducing redundant capacity. PGES controls expert utilization based on the pool shaped by SCEE, improving stability across both previously encountered and unseen tasks. Extensive experiments show that DIMoE-Adapters outperforms previous state-of-the-art methods across various settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。