用视觉问答框架让大模型零样本区分细微图像差异
Zero-Shot Fine-Grained Image Classification Using Large Vision-Language Models
- 将细粒度分类转为视觉问答，避免直接生成类名
- 通过注意力干预提升模型对细微差别的识别能力
- 构建更精准的类别描述数据集，适合研究细粒度识别者
大型视觉语言模型（LVLMs）在视觉-语言推理任务中表现优异，但在零样本细粒度图像分类这一需要精确区分视觉相似类别的挑战性任务中潜力尚未充分挖掘。本文提出一种新方法，将零样本细粒度图像分类转化为视觉问答框架，利用LVLM的综合理解能力，而非依赖直接生成类别名称。通过引入新颖的注意力干预技术提升模型性能，并针对现有数据集的关键局限，构建了更全面、更精确的类别描述基准。在多个细粒度图像分类基准上进行广泛实验验证，结果表明该方法持续优于当前最先进（SOTA）方法，展示了该方法的有效性及LVLM在零样本细粒度分类中的广阔潜力。代码与数据集：https://github.com/Atabuzzaman/Fine-grained-classification
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) have demonstrated impressive performance on vision-language reasoning tasks. However, their potential for zero-shot fine-grained image classification, a challenging task requiring precise differentiation between visually similar categories, remains underexplored. We present a novel method that transforms zero-shot fine-grained image classification into a visual question-answering framework, leveraging LVLMs' comprehensive understanding capabilities rather than relying on direct class name generation. We enhance model performance through a novel attention intervention technique. We also address a key limitation in existing datasets by developing more comprehensive and precise class description benchmarks. We validate the effectiveness of our method through extensive experimentation across multiple fine-grained image classification benchmarks. Our proposed method consistently outperforms the current state-of-the-art (SOTA) approach, demonstrating both the effectiveness of our method and the broader potential of LVLMs for zero-shot fine-grained classification tasks. Code and Datasets: https://github.com/Atabuzzaman/Fine-grained-classification
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。