用外部知识增强CLIP，让模型持续学习新类别时不忘细节特征。
External Knowledge Injection for CLIP-Based Class-Incremental Learning
- 双分支注入框架，从视觉和文本两方面引入外部知识。
- 结合GPT-4重写描述词，提升分类判别力，性能超越现有方法。
- 适合需要长期适应新类别的视觉识别系统，如智能监控、自动驾驶。
类别增量学习（CIL）使系统能持续适应不断变化的数据流。随着预训练的发展，利用预训练的视觉语言模型（如CLIP）为CIL提供了良好起点。然而，CLIP通过将视觉嵌入与类别名称匹配进行决策，忽略了语言所传递的丰富上下文信息。例如，“猫”可分解为尾巴、毛发、面部等特征用于识别。此外，由于模型持续更新，这些细节特征在CIL中会被覆盖，需依赖外部知识补偿。本文提出针对CLIP的外部知识注入框架ENGINE。为增强来自数据集外的知识迁移，设计双分支注入微调框架，分别编码视觉与文本模态的有用信息：视觉分支通过数据增强丰富特征，文本分支利用GPT-4重写判别性描述符。此外，还引入推理阶段的后处理知识重排序机制。注入知识后，模型能更好捕捉随数据演变的有用特征。大量实验表明ENGINE达到当前最优性能。代码已开源：https://github.com/LAMDA-CL/ICCV25-ENGINE
原文摘要 · Abstract (English)
Class-Incremental Learning (CIL) enables learning systems to continuously adapt to evolving data streams. With the advancement of pre-training, leveraging pre-trained vision-language models (e.g., CLIP) offers a promising starting point for CIL. However, CLIP makes decisions by matching visual embeddings to class names, overlooking the rich contextual information conveyed through language. For instance, the concept of ``cat'' can be decomposed into features like tail, fur, and face for recognition. Besides, since the model is continually updated, these detailed features are overwritten in CIL, requiring external knowledge for compensation. In this paper, we introduce ExterNal knowledGe INjEction (ENGINE) for CLIP-based CIL. To enhance knowledge transfer from outside the dataset, we propose a dual-branch injection tuning framework that encodes informative knowledge from both visual and textual modalities. The visual branch is enhanced with data augmentation to enrich the visual features, while the textual branch leverages GPT-4 to rewrite discriminative descriptors. In addition to this on-the-fly knowledge injection, we also implement post-tuning knowledge by re-ranking the prediction results during inference. With the injected knowledge, the model can better capture informative features for downstream tasks as data evolves. Extensive experiments demonstrate the state-of-the-art performance of ENGINE. Code is available at: https://github.com/LAMDA-CL/ICCV25-ENGINE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。