用训练图像替代文本标签，解决CLIP在持续学习中的遗忘问题。
LVP-CLIP:Revisiting CLIP for Continual Learning with Label Vector Pool
- 用训练图像构建标签向量池，替代依赖文本描述的分类方式。
- 在类增量和域增量任务上性能超越当前最优基线40.7%。
- 支持并行学习，计算与内存开销低，适合资源受限场景。
持续学习旨在使模型能够顺序学习新任务而不遗忘旧知识。近期方法常利用视觉-语言模型CLIP，因其高维特征空间和跨模态匹配能力。传统基于CLIP的分类方法通过比较图像与文本嵌入相似性来识别测试图像的类别标签，但对文本短语质量敏感，且在缺乏有意义文本标签的类别上表现较差。本文重新思考基于CLIP的持续学习，提出标签向量池（Label Vector Pool, LVP）概念：用训练图像作为相似性参考，取代文本标签，从而无需理想文本描述。我们设计了三种LVP变体，并在类增量与域增量学习任务上评估其性能。得益于CLIP的高维特征空间，LVP学习算法具有任务顺序无关性，新知识不修改旧知识，几乎无遗忘。不同任务可独立并行学习，计算与内存开销低。实验结果表明，所提LVP方法显著优于当前最先进基线，提升达40.7%。
原文摘要 · Abstract (English)
Continual learning aims to update a model so that it can sequentially learn new tasks without forgetting previously acquired knowledge. Recent continual learning approaches often leverage the vision-language model CLIP for its high-dimensional feature space and cross-modality feature matching. Traditional CLIP-based classification methods identify the most similar text label for a test image by comparing their embeddings. However, these methods are sensitive to the quality of text phrases and less effective for classes lacking meaningful text labels. In this work, we rethink CLIP-based continual learning and introduce the concept of Label Vector Pool (LVP). LVP replaces text labels with training images as similarity references, eliminating the need for ideal text descriptions. We present three variations of LVP and evaluate their performance on class and domain incremental learning tasks. Leveraging CLIP's high dimensional feature space, LVP learning algorithms are task-order invariant. The new knowledge does not modify the old knowledge, hence, there is minimum forgetting. Different tasks can be learned independently and in parallel with low computational and memory demands. Experimental results show that proposed LVP-based methods outperform the current state-of-the-art baseline by a significant margin of 40.7%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。