动态优化提示词,让CLIP模型更懂细节。
AttriPrompt: Dynamic Prompt Composition Learning for CLIP
- 用视觉特征动态检索并拼接提示词,实现内容自适应。
- 在三个基准上提升最多7.37%,尤其擅长跨域迁移。
- 适合需要细粒度理解与泛化能力的视觉语言任务。
提示学习的发展推动了更深层提示设计的探索以提升模型性能。然而现有深度文本提示方法存在两大缺陷:过度依赖对比学习目标,强调高层语义对齐而忽略细粒度特征优化;所有输入类别使用固定提示,缺乏内容感知适应性。为此,我们提出AttriPrompt——一种新框架,通过利用CLIP视觉编码器中间层特征来增强和精炼文本语义表示。设计了属性检索模块,先对每一层视觉特征进行聚类,再基于聚合视觉特征从提示池中检索语义相似提示,并将其拼接到文本编码器每层输入。借助提示文本特征中嵌入的分层视觉信息,引入双流对比学习实现细粒度对齐。此外,通过在提示与非提示文本特征间施加显式正则化约束,引入自正则化机制,防止在有限训练数据上的过拟合。在三个基准上的大量实验表明,AttriPrompt优于现有最优方法,在基类到新类设置下最高提升7.37%。其在跨域知识迁移中的表现突出,使视觉-语言预训练模型更具实际应用潜力。
原文摘要 · Abstract (English)
The evolution of prompt learning methodologies has driven exploration of deeper prompt designs to enhance model performance. However, current deep text prompting approaches suffer from two critical limitations: Over-reliance on constrastive learning objectives that prioritize high-level semantic alignment, neglecting fine-grained feature optimization; Static prompts across all input categories, preventing content-aware adaptation. To address these limitations, we propose AttriPrompt-a novel framework that enhances and refines textual semantic representations by leveraging the intermediate-layer features of CLIP's vision encoder. We designed an Attribute Retrieval module that first clusters visual features from each layer. The aggregated visual features retrieve semantically similar prompts from a prompt pool, which are then concatenated to the input of every layer in the text encoder. Leveraging hierarchical visual information embedded in prompted text features, we introduce Dual-stream Contrastive Learning to realize fine-grained alignment. Furthermore, we introduce a Self-Regularization mechanism by applying explicit regularization constraints between the prompted and non-prompted text features to prevent overfitting on limited training data. Extensive experiments across three benchmarks demonstrate AttriPrompt's superiority over state-of-the-art methods, achieving up to 7.37\% improvement in the base-to-novel setting. The observed strength of our method in cross-domain knowledge transfer positions vision-language pre-trained models as more viable solutions for real-world implementation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。