用生成式提示学习提升视觉模型细粒度识别能力
Towards Generative Class Prompt Learning for Fine-grained Visual Recognition
- 通过可学习提示与扩散模型生成类级特征
- 在少样本场景下显著超越现有方法
- 适合做细粒度图像识别的研究者
尽管基础视觉语言模型在多种语义判别任务中表现优异，但在细粒度分类上仍存在困难。此外，基于单一领域训练的基础模型在跨域场景下泛化能力差。本文认为问题源于视觉语言表征的局限性，提出两种新方法：生成式类别提示学习（GCPL）和对比多类别提示学习（CoMPLe）。GCPL利用文本到图像扩散模型，通过少量样本和可学习类别提示增强类别嵌入的视听协同；CoMPLe在此基础上引入对比学习机制，在生成优化过程中强化类别间分离。实验证明该方法在少样本图像识别任务中显著优于现有技术。
原文摘要 · Abstract (English)
Although foundational vision-language models (VLMs) have proven to be very successful for various semantic discrimination tasks, they still struggle to perform faithfully for fine-grained categorization. Moreover, foundational models trained on one domain do not generalize well on a different domain without fine-tuning. We attribute these to the limitations of the VLM's semantic representations and attempt to improve their fine-grained visual awareness using generative modeling. Specifically, we propose two novel methods: Generative Class Prompt Learning (GCPL) and Contrastive Multi-class Prompt Learning (CoMPLe). Utilizing text-to-image diffusion models, GCPL significantly improves the visio-linguistic synergy in class embeddings by conditioning on few-shot exemplars with learnable class prompts. CoMPLe builds on this foundation by introducing a contrastive learning component that encourages inter-class separation during the generative optimization process. Our empirical results demonstrate that such a generative class prompt learning approach substantially outperform existing methods, offering a better alternative to few shot image recognition challenges. The source code will be made available at: https://github.com/soumitri2001/GCPL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。