提升视觉语言模型泛化能力,让模型在新类别和未知领域表现更稳定。
DiSa: Directional Saliency-Aware Prompt Learning for Generalizable Vision-Language Models
- 通过注意力引导聚焦关键图像区域,减少对无关片段的依赖。
- 利用类别原型方向对齐,保持特征一致性,提升跨域适应性。
- 在11个基准上超越现有方法,尤其适用于少样本和跨数据集场景。
提示学习已成为适配视觉语言模型(如CLIP)至下游任务的强大范式。然而,现有方法常对已见数据过拟合,导致在新类别或未见领域上性能显著下降。为此,我们提出DiSa,一种方向敏感性感知的提示学习框架,融合两种互补的正则化策略以增强泛化能力。首先,交叉交互正则化(CIR)通过促使提示编码器与冻结编码器协同学习,促进跨模态对齐;在CIR中,基于显著性的掩码策略引导图像编码器优先关注语义关键区域,降低对不相关信息块的依赖。其次,我们引入方向正则化策略,以方向性方式对齐视觉嵌入与类别原型特征,强调特征方向的一致性而非严格距离接近,从而利用类别均值统计得出的稳定原型方向实现鲁棒泛化。在11个多样化的图像分类基准上的广泛评估表明,DiSa在多种设置下持续优于当前最优提示学习方法,包括基础到新类泛化、跨数据集迁移、领域泛化和少样本学习。
原文摘要 · Abstract (English)
Prompt learning has emerged as a powerful paradigm for adapting vision-language models such as CLIP to downstream tasks. However, existing methods often overfit to seen data, leading to significant performance degradation when generalizing to novel classes or unseen domains. To address this limitation, we propose DiSa, a Directional Saliency-Aware Prompt Learning framework that integrates two complementary regularization strategies to enhance generalization. First, our Cross-Interactive Regularization (CIR) fosters cross-modal alignment by enabling cooperative learning between prompted and frozen encoders. Within CIR, a saliency-aware masking strategy guides the image encoder to prioritize semantically critical image regions, reducing reliance on less informative patches. Second, we introduce a directional regularization strategy that aligns visual embeddings with class-wise prototype features in a directional manner to prioritize consistency in feature orientation over strict proximity. This approach ensures robust generalization by leveraging stable prototype directions derived from class-mean statistics. Extensive evaluations on 11 diverse image classification benchmarks demonstrate that DiSa consistently outperforms state-of-the-art prompt learning methods across various settings, including base-to-novel generalization, cross-dataset transfer, domain generalization, and few-shot learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。