提出AFA框架，让视觉语言模型持续学习新任务时保持并提升零样本识别能力。
Beyond CLIP Generalization: Against Forward&Backward Forgetting Adapter for Continual Learning of Vision-Language Models
- 设计双向遗忘抑制适配器，分别增强零样本与少样本学习能力
- 在少样本增量学习中显著超越CLIP基线，迁移性能更优
- 适合需要长期更新且保持泛化能力的视觉语言模型应用
本研究针对多领域任务增量学习（MTIL）问题，要求视觉语言模型（VLMs）在持续学习新知识的同时维持其固有的零样本识别能力。现有方法将未见领域样本的测试任务交由原始CLIP处理，仅防止模型零样本能力下降，却无法进一步提升VLM的泛化性能。为此，本文提出新型MTIL框架AFA，包含两个核心模块：(1) 反前向遗忘适配器，为每个增量任务的数据集学习任务无关信息，以增强VLM的零样本识别能力；(2) 反后向遗忘适配器，强化VLM的少样本学习能力，同时支持增量学习。大量实验表明，AFA方法在少样本MTIL任务中显著优于现有最先进方法，并在可迁移性上超越CLIP的原始零样本性能。代码见附录。
原文摘要 · Abstract (English)
This study aims to address the problem of multi-domain task incremental learning~(MTIL), which requires that vision-language models~(VLMs) continuously acquire new knowledge while maintaining their inherent zero-shot recognition capability. Existing paradigms delegate the testing of unseen-domain samples to the original CLIP, which only prevents the degradation of the model's zero-shot capability but fails to enhance the generalization of the VLM further. To this end, we propose a novel MTIL framework, named AFA, which comprises two core modules: (1) an against forward-forgetting adapter that learns task-invariant information for each dataset in the incremental tasks to enhance the zero-shot recognition ability of VLMs; (2) an against backward-forgetting adapter that strengthens the few-shot learning capability of VLMs while supporting incremental learning. Extensive experiments demonstrate that the AFA method significantly outperforms existing state-of-the-art approaches, especially in few-shot MTIL tasks, and surpasses the inherent zero-shot performance of CLIP in terms of transferability. The code is provided in the Supplementary Material.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。