探索视觉语言模型的纯视觉分类能力，发现图文互补并提出无需训练的融合方法。
Image Recognition with Vision and Language Embeddings of VLMs
- 用图文双编码器模型对比图文引导与纯视觉分类效果。
- 在ImageNet-1k上实现85.3%准确率，视觉相似性对部分类别更有效。
- 提出按类别精度融合策略，适合提升零样本图像识别性能。
视觉语言模型（VLMs）通过图像-文本对齐实现了强大的零样本分类能力，但其纯视觉推理能力仍缺乏深入研究。本文系统评估了多种双编码器VLMs（包括SigLIP 2和RADIOv2.5）在图像-文本引导和纯视觉分类下的表现，基于标准设置在ImageNet-1k验证集及其标签修正版本上进行对比。分析了提示设计、类别多样性、k-NN邻居数量和参考集大小对准确率的影响。结果显示，语言与视觉具有互补优势：某些类别更适合文本提示，而另一些则依赖视觉相似性。为利用这种互补性，我们提出一种无需学习的融合方法，基于每类精度加权融合，显著提升分类性能。代码已公开于https://github.com/gonikisgo/bmvc2025-vlm-image-recognition。
原文摘要 · Abstract (English)
Vision-language models (VLMs) have enabled strong zero-shot classification through image-text alignment. Yet, their purely visual inference capabilities remain under-explored. In this work, we conduct a comprehensive evaluation of both language-guided and vision-only image classification with a diverse set of dual-encoder VLMs, including both well-established and recent models such as SigLIP 2 and RADIOv2.5. The performance is compared in a standard setup on the ImageNet-1k validation set and its label-corrected variant. The key factors affecting accuracy are analysed, including prompt design, class diversity, the number of neighbours in k-NN, and reference set size. We show that language and vision offer complementary strengths, with some classes favouring textual prompts and others better handled by visual similarity. To exploit this complementarity, we introduce a simple, learning-free fusion method based on per-class precision that improves classification performance. The code is available at: https://github.com/gonikisgo/bmvc2025-vlm-image-recognition.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。