解决CLIP模型持续学习时跨模态不一致问题,提升新类别识别准确率。
Preserving Cross-Modal Consistency for CLIP-based Class-Incremental Learning
- 分两阶段解耦视觉编码器与文本软提示优化,保持跨模态对齐
- 在多个数据集上达到当前最优,平均准确率提升1.80%
- 适合研究持续学习与多模态模型融合的开发者参考
类别增量学习(CIL)使模型能从连续任务中不断学习新类别,同时避免遗忘旧知识。尽管视觉-语言模型如CLIP在跨域泛化方面表现优异,但将其扩展到持续学习场景仍具挑战。特别是为新类别学习任务特定的软提示时,因缺乏历史数据,文本原型容易过拟合于近期类别,导致分类器偏差。本文提出DMC,一种简单有效的两阶段框架,将视觉编码器适应与文本软提示优化解耦,各阶段训练时冻结另一模态,使一模态作为稳定语义锚点以维持跨模态一致性。此外,现有方法通常用类级高斯统计进行生成回放,却忽视了视觉编码器更新带来的分布漂移。为此,我们引入DMC-OT,采用最优传输引导校准策略,对演化编码器间的记忆统计进行对齐,并设计任务特异性提示以增强跨任务可分性。在CIFAR-100、ImageNet-R、CUB-200和UCF-101上的大量实验表明,两者均达到当前最优性能,其中DMC-OT平均准确率进一步提升1.80%。
原文摘要 · Abstract (English)
Class-incremental learning (CIL) enables models to continuously learn new categories from sequential tasks without forgetting previously acquired knowledge. While recent advances in vision-language models such as CLIP have demonstrated strong generalization across domains, extending them to continual settings remains challenging. In particular, learning task-specific soft prompts for newly introduced classes often leads to severe classifier bias, as the text prototypes overfit to recent categories when prior data are unavailable. In this paper, we propose DMC, a simple yet effective two-stage framework for CLIP-based CIL that decouples the adaptation of the vision encoder and the optimization of textual soft prompts. Each stage is trained with the other frozen, allowing one modality to act as a stable semantic anchor for the other to preserve cross-modal alignment. Furthermore, current CLIP-based CIL approaches typically store class-wise Gaussian statistics for generative replay, yet they overlook the distributional drift that arises when the vision encoder is updated over time. To address this issue, we introduce DMC-OT, an enhanced version of DMC that incorporates an optimal-transport guided calibration strategy to align memory statistics across evolving encoders, along with a task-specific prompting design that enhances inter-task separability. Extensive experiments on CIFAR-100, Imagenet-R, CUB-200, and UCF-101 demonstrate that both DMC and DMC-OT achieve state-of-the-art performance, with DMC-OT further improving accuracy by an average of 1.80%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。