通过模型融合缓解视觉语言模型的灾难性遗忘问题。
Enhanced Continual Learning of Vision-Language Models with Model Fusion
- 采用解耦-统一机制，动态融合旧任务专家与新任务专家。
- 在MTIL基准上平均性能提升2%，零样本能力优于原始模型。
- 无需额外参考数据，适合参数高效微调场景的持续学习。
视觉语言模型（VLMs）通过融合视觉与文本模态，在零样本任务中表现出色，但顺序微调多个下游任务时易发生灾难性遗忘。现有持续学习方法常依赖额外参考数据、牺牲零样本性能或仅适用于参数高效微调。本文提出一种新型持续解耦-统一（ConDU）方法，首次将模型融合应用于VLM的持续学习。ConDU维护一个统一模型及任务触发器和原型集，通过迭代解耦旧任务专家并将其与新任务专家统一。此外，引入多解耦专家预测聚合的推理策略以支持零样本场景。在MTIL基准上的大量实验表明，ConDU相比最先进基线在所有已见任务上平均性能提升2%，同时增强零样本能力。代码已开源。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) represent a significant breakthrough in artificial intelligence by integrating visual and textual modalities to achieve impressive zero-shot capabilities. However, VLMs are susceptible to catastrophic forgetting when sequentially fine-tuned on multiple downstream tasks. Existing continual learning methods for VLMs face various limitations, often relying on additional reference datasets, compromising zero-shot performance, or being restricted to parameter-efficient fine-tuning scenarios. In this paper, we propose a novel Continual Decoupling-Unifying (ConDU) approach that pioneers the use of model fusion for continual learning in VLMs. Specifically, ConDU maintains a unified model along with task triggers and prototype sets, employing an iterative process of decoupling task experts for previous tasks and unifying them with the task expert for the newly learned task. Additionally, we introduce an inference strategy for zero-shot scenarios by aggregating predictions from multiple decoupled task experts. Extensive experiments on the MTIL benchmark show that ConDU achieves up to a 2\% improvement in average performance across all seen tasks compared to state-of-the-art baselines, while also enhancing zero-shot capabilities relative to the original VLM. Our code is available at https://github.com/zhangzicong518/ConDU.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。