用多模态大模型做图像检索,无需训练就能提升精度。
Indexing Multimodal Language Models for Large-scale Image Retrieval

- 用配对图像提示模型,通过下一个词概率算相似度。
- 在多个数据集上超越专用重排序模型,抗干扰能力强。
- 适合开放世界的大规模图像检索,尤其适合没标注数据的场景。
多模态大语言模型(MLLMs)展现出强大的跨模态推理能力,但其在纯视觉任务中的潜力尚未充分挖掘。本文探索将MLLMs作为无训练的实例级图像到图像检索相似度估计器。通过向模型输入成对图像,并将下一个词的概率转换为相似度分数,实现零样本重排序,集成至大规模检索流程中。该方法无需专用架构或微调,充分利用多模态预训练中学习到的丰富视觉区分能力。为提升可扩展性,结合内存高效的索引与top-k候选重排序策略。在多个基准测试中,MLLMs在非原生领域表现优于专用重排序器,且对杂乱、遮挡和小物体具有更强鲁棒性。尽管效果显著,但在严重外观变化下仍存在失效情况,揭示了未来研究空间。结果表明,MLLMs是开放世界大规模图像检索的一种有前景替代方案。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have demonstrated strong cross-modal reasoning capabilities, yet their potential for vision-only tasks remains underexplored. We investigate MLLMs as training-free similarity estimators for instance-level image-to-image retrieval. Our approach prompts the model with paired images and converts next-token probabilities into similarity scores, enabling zero-shot re-ranking within large-scale retrieval pipelines. This design avoids specialized architectures and fine-tuning, leveraging the rich visual discrimination learned during multimodal pre-training. We address scalability by combining MLLMs with memory-efficient indexing and top-$k$ candidate re-ranking. Experiments across diverse benchmarks show that MLLMs outperform task-specific re-rankers outside their native domains and exhibit superior robustness to clutter, occlusion, and small objects. Despite strong results, we identify failure modes under severe appearance changes, highlighting opportunities for future research. Our findings position MLLMs as a promising alternative for open-world large-scale image retrieval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。