让CLIP模型学会利用局部图像特征,提升增量学习效果。
Unlocking Patch-Level Features for CLIP-Based Class-Incremental Learning

- 通过语义引导选择关键视觉片段,实现细粒度对齐
- 在多个基准上超越现有方法,显著降低遗忘率
- 适合关注视觉细节与长期学习的开发者
类增量学习(CIL)使模型能持续学习新知识并缓解灾难性遗忘。受CLIP卓越泛化能力推动,基于预训练视觉-语言模型的CIL已成为主流范式。然而,现有方法主要聚焦于全局图像嵌入(如[CLS]标记)与对应文本提示(如[EOS]标记)的对齐,忽略了CLIP编码器中蕴含的丰富局部片段语义信息。例如识别兔子时,局部图像块可能编码长耳朵、蓬松尾巴等独特线索,可作为识别补充证据。针对此问题,本文提出语义引导的局部对齐方法(SPA),旨在激活CLIP中被忽视的局部表征。具体而言,为每类构造代表性且多样的视觉样本,输入GPT-5生成类别级语义描述,用以指导判别性局部特征的选择。在此基础上,采用最优传输对齐选定图像块与类别描述中的语义标记,实现结构化跨模态对齐,提升识别能力。此外,引入任务特定投影器以适应下游增量任务,并从存储的类别高斯统计量中采样伪特征,校准旧类别表示,有效缓解遗忘。大量实验表明,SPA达到当前最优性能。
原文摘要 · Abstract (English)
Class-Incremental Learning (CIL) enables models to continuously integrate new knowledge while mitigating catastrophic forgetting. Driven by the remarkable generalization of CLIP, leveraging pre-trained vision-language models has become a dominant paradigm in CIL. However, current work primarily focuses on aligning global image embeddings (i.e., [CLS] token) with their corresponding text prompts (i.e., [EOS] token). Despite their good performance, we find that they discard the rich patch-level semantic information inherent in CLIP's encoders. For instance, when recognizing a rabbit, local patches may encode its distinctive cues, such as long ears and a fluffy tail, which can provide complementary evidence for recognition. Based on the above observation, we propose SPA (Semantic-guided Patch-level Alignment) for CLIP-based CIL, which aims to awaken long-neglected local representations within CLIP. Specifically, for each class, we first construct representative and diverse visual samples and feed them to GPT-5 as visual guidance to generate class-wise semantic descriptions. These descriptions are used to guide the selection of discriminative patch-level visual features. Building upon these selected patches, we further employ optimal transport to align selected patch tokens with semantic tokens from class-wise descriptions, yielding a structured cross-modal alignment that improves recognition. Furthermore, we introduce task-specific projectors for effective adaptation to downstream incremental tasks, and sample pseudo-features from stored class-wise Gaussian statistics to calibrate old-class representations, thereby mitigating catastrophic forgetting. Extensive experiments demonstrate that SPA achieves state-of-the-art performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。