无需预设词汇表,用语言增强视觉模型实现细粒度图像识别
Vocabulary-free Fine-grained Visual Recognition via Enriched Contextually Grounded Vision-Language Model
- 利用语言上下文增强的视觉语言模型,摆脱固定标签限制
- 在多个数据集上达到当前最优性能,且无需训练
- 适合无标注新类别场景,适合零样本/少样本应用
细粒度图像分类旨在区分同一大类中视觉相似的子类别(如鸟种、车型、花种),是计算机视觉中的挑战性任务。传统方法依赖固定词汇表和闭集分类范式,难以适应现实世界中新类别频繁出现的情况。近期研究显示,将大语言模型(LLMs)与视觉语言模型(VLMs)结合可实现开集识别,无需预定义类别标签。然而,现有方法在分类阶段未能充分挖掘LLM潜力,且依赖未经验证的LLM生成类别名。为此,我们提出无需训练的方法Enriched-FineR(E-FineR),在细粒度视觉识别中表现领先,同时具备更强可解释性,适用于专家标注困难的真实场景和新领域。此外,该方法在零样本与少样本分类中性能媲美现有最先进水平,且无需训练、无需人工干预。整体上,我们的词汇表无关框架推动图像分类从刚性标签预测转向灵活的语言驱动理解,支持可扩展、泛化性强的真实应用系统。代码已公开于https://github.com/demidovd98/e-finer。
原文摘要 · Abstract (English)
Fine-grained image classification, the task of distinguishing between visually similar subcategories within a broader category (e.g., bird species, car models, flower types), is a challenging computer vision problem. Traditional approaches rely heavily on fixed vocabularies and closed-set classification paradigms, limiting their scalability and adaptability in real-world settings where novel classes frequently emerge. Recent research has demonstrated that combining large language models (LLMs) with vision-language models (VLMs) makes open-set recognition possible without the need for predefined class labels. However, the existing methods are often limited in harnessing the power of LLMs at the classification phase, and also rely heavily on the guessed class names provided by an LLM without thorough analysis and refinement. To address these bottlenecks, we propose our training-free method, Enriched-FineR (or E-FineR for short), which demonstrates state-of-the-art results in fine-grained visual recognition while also offering greater interpretability, highlighting its strong potential in real-world scenarios and new domains where expert annotations are difficult to obtain. Additionally, we demonstrate the application of our proposed approach to zero-shot and few-shot classification, where it demonstrated performance on par with the existing SOTA while being training-free and not requiring human interventions. Overall, our vocabulary-free framework supports the shift in image classification from rigid label prediction to flexible, language-driven understanding, enabling scalable and generalizable systems for real-world applications. Well-documented code is available on https://github.com/demidovd98/e-finer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。