为提示学习注入类别知识,提升视觉语言模型的分类精度。
Plug-and-play Class-aware Knowledge Injection for Prompt Learning with Visual-Language Model

- 通过类别特定提示生成与查询-键匹配机制,融合类别级知识。
- 在基类和新类别上均显著提升现有方法性能。
- 可插拔设计,适用于各类提示学习框架,适合零样本分类任务。
提示学习已成为提升视觉语言模型(如CLIP)在下游任务中表现的有效方法,尤其在特定领域的零样本分类中广泛应用。现有方法通常聚焦于学习共享类别的提示或通过条件提示学习生成实例特定提示。然而,这些方法常忽略提示设计中的类别特定知识,导致性能受限。主要原因在于:1)类别特定提示提供更细粒度的监督,有助于防止不同类别的数据被误归为同一类;2)相较于类别特定提示,实例特定提示忽略了多个实例间的类别级信息,可能导致同一类别的数据被划分为多个类别。为此,我们提出一种即插即用的类别感知知识注入(CAKI)框架。CAKI包含两个核心组件:类别特定提示生成与查询-键提示匹配。前者将同类别少量样本中的类别知识编码至提示,并存入类别级知识库;后者为每个测试实例提供从知识库中检索相关类别知识并注入以优化预测的插件式机制。大量实验表明,本方法有效提升了现有方法在基础类和新类别上的性能。代码已公开于 https://github.com/yjh576/CAKI。
原文摘要 · Abstract (English)
Prompt learning has become an effective and widely used technique in enhancing vision-language models (VLMs) such as CLIP for various downstream tasks, particularly in zero-shot classification within specific domains. Existing methods typically focus on either learning class-shared prompts for a given domain or generating instance-specific prompts through conditional prompt learning. While these methods have achieved promising performance, they often overlook class-specific knowledge in prompt design, leading to suboptimal outcomes. The underlying reasons are: 1) class-specific prompts offer more fine-grained supervision compared to coarse class-shared prompts, which helps prevent misclassification of data from different classes into a single class; 2) compared to class-specific prompts, instance-specific prompts neglect the richer class-level information across multiple instances, potentially causing data from the same class to be divided into multiple classes. To effectively supplement the class-specific knowledge into existing methods, we propose a plug-and-play Class-Aware Knowledge Injection (CAKI) framework. CAKI comprises two key components, i.e., class-specific prompt generation and query-key prompt matching. The former encodes class-specific knowledge into prompts from few-shot samples that belong to the same class and stores the learned prompts in a class-level knowledge bank. The latter provides a plug-and-play mechanism for each test instance to retrieve relevant class-level knowledge from the knowledge bank and inject such knowledge to refine model predictions. Extensive experiments demonstrate that our CAKI effectively improves the performance of existing methods on base and novel classes. Code is publicly available at \href{https://github.com/yjh576/CAKI}{this https URL}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。