用大模型生成标签,让小模型高效完成细粒度图像识别。
Efficient Vocabulary-Free Fine-Grained Visual Recognition in the Age of Multimodal LLMs
- 用多模态大模型生成无预设标签的图像标签,构建弱监督数据集。
- 在不依赖人工标注的情况下,实现接近有监督效果的细粒度识别。
- 适合医疗等无标签数据场景,推理成本低,适合实际部署。
细粒度视觉识别(FGVR)需区分外观极其相似的类别,因类间差异细微且需大量专家标注数据而极具挑战。在医学影像等场景中,由于隐私和标注成本问题,缺乏高质量标注数据。此时,模型无法依赖预定义标签集,预测输出空间为开放形式,称为无词汇表细粒度视觉识别(VF-FGVR)。尽管多模态大语言模型(MLLM)在该任务上有潜力,但对每个测试样本调用大模型代价高昂、推理延迟高。为此,我们提出近邻标签精炼(NeaR),通过用MLLM生成标签来微调下游CLIP模型。该方法从少量未标注训练数据出发,利用MLLM生成标签构建弱监督数据集。NeaR能有效处理MLLM标签的噪声、随机性和开放性,显著提升效率并建立新基准。
原文摘要 · Abstract (English)
Fine-grained Visual Recognition (FGVR) involves distinguishing between visually similar categories, which is inherently challenging due to subtle inter-class differences and the need for large, expert-annotated datasets. In domains like medical imaging, such curated datasets are unavailable due to issues like privacy concerns and high annotation costs. In such scenarios lacking labeled data, an FGVR model cannot rely on a predefined set of training labels, and hence has an unconstrained output space for predictions. We refer to this task as Vocabulary-Free FGVR (VF-FGVR), where a model must predict labels from an unconstrained output space without prior label information. While recent Multimodal Large Language Models (MLLMs) show potential for VF-FGVR, querying these models for each test input is impractical because of high costs and prohibitive inference times. To address these limitations, we introduce \textbf{Nea}rest-Neighbor Label \textbf{R}efinement (NeaR), a novel approach that fine-tunes a downstream CLIP model using labels generated by an MLLM. Our approach constructs a weakly supervised dataset from a small, unlabeled training set, leveraging MLLMs for label generation. NeaR is designed to handle the noise, stochasticity, and open-endedness inherent in labels generated by MLLMs, and establishes a new benchmark for efficient VF-FGVR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。