arXiv:2412.09442cs.CV2024-12ICCV被引 29

用通用属性增强文本提示,让模型识别未知类别。

Advancing Textual Prompt Learning with Anchored Attributes

  • 将软提示从类别维度扩展到多维属性维度,引入可学习属性标记。
  • 在11个数据集上验证,显著提升对未知类别的识别能力。
  • 无需复杂改造,可无缝嵌入现有方法,计算开销极低。

基于文本的提示学习方法通常采用可学习的软提示与硬类别标记级联作为文本输入,旨在对齐图像与文本(类别)空间以支持下游任务。然而,当前训练仅限于已知类别的对齐,无法关联未知类别。为此,本文提出利用通用属性作为桥梁,增强图像与未知类别间的对齐。具体地,我们设计了属性锚定的文本提示学习方法ATPrompt,通过在可学习软提示中融入多个属性标记,将软提示的学习空间从单一类别维度拓展至多维属性维度,实现从类别中心向属性-类别混合形式的转变。此外,我们提出一种简单可微的属性搜索方法,自动识别适合下游任务的代表性属性。作为即插即用的技术,ATPrompt可无缝替换现有文本提示方法的基础格式,在计算成本几乎不变的前提下带来普遍性能提升。大量实验在11个数据集上验证了其有效性。代码已公开于https://github.com/zhengli97/ATPrompt。

原文摘要 · Abstract (English)

Textual-based prompt learning methods primarily employ multiple learnable soft prompts and hard class tokens in a cascading manner as text inputs, aiming to align image and text (category) spaces for downstream tasks. However, current training is restricted to aligning images with predefined known categories and cannot be associated with unknown categories. In this work, we propose utilizing universal attributes as a bridge to enhance the alignment between images and unknown categories. Specifically, we introduce an Attribute-anchored Textual Prompt learning method for vision-language models, named ATPrompt. This approach expands the learning space of soft prompts from the original one-dimensional category level into the multi-dimensional attribute level by incorporating multiple attribute tokens into the learnable soft prompts. Through this modification, we transform the text prompt from a category-centric form to an attribute-category hybrid form. Additionally, we introduce a straightforward differentiable attribute search method to identify representative and suitable attributes for downstream tasks. As an easy-to-use plug-in technique, ATPrompt can seamlessly replace the existing basic prompt format in textual-based methods, providing general improvements at a negligible computational cost. Extensive experiments across 11 datasets validate the effectiveness of our method. Code is publicly available at https://github.com/zhengli97/ATPrompt.

提示学习属性建模零样本识别

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。