arXiv:2411.16018cs.CV2024-11中稿 · IEEE/CVF Winter Co…被引 6

通过风格引导提示学习,提升视觉语言模型在新场景下的泛化能力。

Style-Pro: Style-Guided Prompt Learning for Generalizable Vision-Language Models

  • 用可学习的风格基底生成多样化分布偏移,避免过拟合。
  • 在11个数据集上超越现有方法,尤其在跨域迁移中表现优异。
  • 适合需要强泛化能力的下游任务,如未知类别识别与领域自适应。

预训练的视觉-语言(VL)模型(如CLIP)在下游任务中表现出显著的泛化能力,即使经过极少微调。尽管提示学习已成为适配预训练VL模型的有效策略,但现有方法常因过度依赖特定下游数据分布而产生严重过拟合,限制了模型在新领域或未见类上的泛化能力。为解决此问题,本文提出Style-Pro,一种新型风格引导的提示学习框架,有效缓解过拟合并保留CLIP的零样本泛化能力。Style-Pro采用可学习的风格基底,结合两种专用损失函数,合成多样化的分布偏移,同时确保风格多样性与内容完整性。为减少未见领域与源域之间的差异,该方法将未见风格映射为已知风格基底的加权组合。此外,为保持风格调整后的提示模型与原始冻结的CLIP之间的一致性,引入一致性约束,最小化嵌入空间中的偏差。在11个基准数据集上的大量实验表明,Style-Pro在基础到新类泛化、跨数据集迁移和领域泛化等多种设置下均持续优于当前最优方法。

原文摘要 · Abstract (English)

Pre-trained Vision-language (VL) models, such as CLIP, have shown significant generalization ability to downstream tasks, even with minimal fine-tuning. While prompt learning has emerged as an effective strategy to adapt pre-trained VL models for downstream tasks, current approaches frequently encounter severe overfitting to specific downstream data distributions. This overfitting constrains the original behavior of the VL models to generalize to new domains or unseen classes, posing a critical challenge in enhancing the adaptability and generalization of VL models. To address this limitation, we propose Style-Pro, a novel style-guided prompt learning framework that mitigates overfitting and preserves the zero-shot generalization capabilities of CLIP. Style-Pro employs learnable style bases to synthesize diverse distribution shifts, guided by two specialized loss functions that ensure style diversity and content integrity. Then, to minimize discrepancies between unseen domains and the source domain, Style-Pro maps the unseen styles into the known style representation space as a weighted combination of style bases. Moreover, to maintain consistency between the style-shifted prompted model and the original frozen CLIP, Style-Pro introduces consistency constraints to preserve alignment in the learned embeddings, minimizing deviation during adaptation to downstream tasks. Extensive experiments across 11 benchmark datasets demonstrate the effectiveness of Style-Pro, consistently surpassing state-of-the-art methods in various settings, including base-to-new generalization, cross-dataset transfer, and domain generalization.

视觉语言模型提示学习泛化能力风格引导

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。