用无标签数据自动优化提示词，提升大模型细粒度图像分类能力
Unlabeled Data Improves Fine-Grained Image Zero-shot Classification with Multimodal LLMs
- 基于无标签数据迭代生成引导提示词，无需训练模型
- 在多个数据集上平均比标准零样本方法提升13%
- 适合想提升大模型细粒度识别能力的研究者使用
尽管多模态大语言模型（MLLMs）在通用零样本图像分类任务中表现良好，但细粒度分类仍具挑战性，需精准捕捉细微视觉差异。为解决此问题，我们提出AutoSEP——一种完全无监督的迭代自监督提示学习框架，利用无标签数据学习能引导MLLM识别关键判别特征的描述提示词，从而提升分类准确率。该框架基于实例级分类评分函数，通过迭代优化提示词，仅需黑箱访问MLLM，无需任何训练或微调。我们在多个细粒度分类数据集上评估该方法，结果表明其持续优于其他无监督基线，平均较标准零样本分类提升13%，较最优基线提升5%。代码已开源。
原文摘要 · Abstract (English)
Despite Multimodal Large Language Models (MLLMs) showing promising results on general zero-shot image classification tasks, fine-grained image classification remains challenging. It demands precise attention to subtle visual details to distinguish between visually similar subcategories--details that MLLMs may easily overlook without explicit guidance. To address this, we introduce AutoSEP, an iterative self-supervised prompt learning framework designed to enhance MLLM fine-grained classification capabilities in a fully unsupervised manner. Our core idea is to leverage unlabeled data to learn a description prompt that guides MLLMs in identifying crucial discriminative features within an image, and boosts classification accuracy. We developed an automatic self-enhancing prompt learning framework called AutoSEP to iteratively improve the description prompt using unlabeled data, based on instance-level classification scoring function. AutoSEP only requires black-box access to MLLMs, eliminating the need for any training or fine-tuning. We evaluate our approach on multiple fine-grained classification datasets. It consistently outperforms other unsupervised baselines, demonstrating the effectiveness of our self-supervised optimization framework. Notably, AutoSEP on average improves 13 percent over standard zero-shot classification and 5 percent over the best-performing baselines. Code is available at: https://github.com/yq-hong/AutoSEP
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。