用视觉模型内部特征优化文本提示，提升跨类别识别效果
TextRefiner: Internal Visual Feature as Efficient Refiner for Vision-Language Models Prompt Tuning
- 利用图像分支局部令牌提取细粒度视觉特征，构建本地缓存
- 在11个基准上将CoOp准确率从71.66%提升至76.94%
- 无需外部语言模型，适合高效部署的提示调优场景
尽管提示学习在将视觉-语言模型（VLMs）迁移到下游任务中表现出高效性，但现有方法主要以粗粒度方式学习提示，即所有类别共享相同的提示向量。这导致提示难以区分具有相似或复杂视觉属性的类别，从而限制了迁移性能。近期工作通过引入大语言模型（LLMs）的外部知识来提供类别描述以缓解此问题，但带来了显著的推理开销。本文提出TextRefiner，一种即插即用的方法，通过利用VLM自身的内部知识来优化现有方法的文本提示。具体而言，TextRefiner设计了一个新型局部缓存模块，从图像分支的局部令牌中提取细粒度视觉概念，并将其与文本分支的原始输出聚合对齐，从而高效地精炼和丰富已有提示。例如，在11个基准上，TextRefiner将CoOp的性能从71.66%提升至76.94%，超过引入实例级特征的CoCoOp。结合TextRefiner后，PromptKD达到当前最优性能且推理效率高。代码已开源：https://github.com/xjjxmu/TextRefiner。
原文摘要 · Abstract (English)
Despite the efficiency of prompt learning in transferring vision-language models (VLMs) to downstream tasks, existing methods mainly learn the prompts in a coarse-grained manner where the learned prompt vectors are shared across all categories. Consequently, the tailored prompts often fail to discern class-specific visual concepts, thereby hindering the transferred performance for classes that share similar or complex visual attributes. Recent advances mitigate this challenge by leveraging external knowledge from Large Language Models (LLMs) to furnish class descriptions, yet incurring notable inference costs. In this paper, we introduce TextRefiner, a plug-and-play method to refine the text prompts of existing methods by leveraging the internal knowledge of VLMs. Particularly, TextRefiner builds a novel local cache module to encapsulate fine-grained visual concepts derivedfrom local tokens within the image branch. By aggregating and aligning the cached visual descriptions with the original output of the text branch, TextRefiner can efficiently refine and enrich the learned prompts from existing methods without relying on any external expertise. For example, it improves the performance of CoOp from 71.66 % to 76.94 % on 11 benchmarks, surpassing CoCoOp which introduces instance-wise features for text prompts. Equipped with TextRefiner, PromptKD achieves state-of-the-art performance and is efficient in inference. Our code is relesed at https://github.com/xjjxmu/TextRefiner
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。