arXiv:2508.04136cs.CVcs.AI2025-08被引 1

无需训练,用图文检索实现细粒度图像分类,效果超越多数现有方法。

UniFGVC: Universal Training-Free Few-Shot Fine-Grained Vision Classification via Attribute-Aware Multimodal Retrieval

  • 将少样本细粒度分类转化为图文检索任务,利用大模型生成精准属性描述。
  • 在12个基准上均优于基于CLIP的少样本方法,部分接近全监督大模型表现。
  • 适用于多种视觉语言模型和编码器,通用性强且无过拟合风险。

少样本细粒度视觉分类(FGVC)旨在用少量数据让模型区分细微差异类别。现有方法多依赖微调预训练视觉语言模型,但存在过拟合和泛化能力弱的问题。为此,本文提出UniFGVC,一种通用的免训练框架,将少样本FGVC重构成多模态检索任务。首先,提出类别判别性视觉描述生成器(CDV-Captioner),利用多模态大语言模型的开放世界知识,通过思维链提示和视觉相似参考图生成结构化文本描述,捕捉类间细微属性差异,减少幻觉并增强区分度。由此可将每张图像转为图文对,构建少样本支持集的多模态类别模板。随后,使用现成的视觉与文本编码器嵌入查询与模板,在联合空间中通过最近邻检索完成分类。UniFGVC兼容多种MLLMs与编码器,具备良好泛化性与适应性。在12个FGVC基准上的大量实验表明,其性能持续优于先前基于CLIP的少样本方法,甚至超越部分全监督基于MLLMs的方法。

原文摘要 · Abstract (English)

Few-shot fine-grained visual classification (FGVC) aims to leverage limited data to enable models to discriminate subtly distinct categories. Recent works mostly finetuned the pre-trained visual language models to achieve performance gain, yet suffering from overfitting and weak generalization. To deal with this, we introduce UniFGVC, a universal training-free framework that reformulates few-shot FGVC as multimodal retrieval. First, we propose the Category-Discriminative Visual Captioner (CDV-Captioner) to exploit the open-world knowledge of multimodal large language models (MLLMs) to generate a structured text description that captures the fine-grained attribute features distinguishing closely related classes. CDV-Captioner uses chain-of-thought prompting and visually similar reference images to reduce hallucination and enhance discrimination of generated captions. Using it we can convert each image into an image-description pair, enabling more comprehensive feature representation, and construct the multimodal category templates using few-shot samples for the subsequent retrieval pipeline. Then, off-the-shelf vision and text encoders embed query and template pairs, and FGVC is accomplished by retrieving the nearest template in the joint space. UniFGVC ensures broad compatibility with diverse MLLMs and encoders, offering reliable generalization and adaptability across few-shot FGVC scenarios. Extensive experiments on 12 FGVC benchmarks demonstrate its consistent superiority over prior few-shot CLIP-based methods and even several fully-supervised MLLMs-based approaches.

少样本学习细粒度分类多模态检索免训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。