用大模型推理重排图像,让检索更准更可解释。
Chain-of-Thought Re-ranking for Image Retrieval Tasks
- 让大模型直接参与排序,进行全局对比推理
- 在5个数据集上达到当前最佳效果
- 适合需要高精度与可解释性的图像检索场景
图像检索是计算机视觉中的基础且挑战性问题。尽管多模态大语言模型(MLLM)展现出强大的推理能力,现有方法通常仅将其用于评估,未让其直接参与重排过程,导致其丰富的多模态推理能力未被充分利用,性能受限。本文提出一种新的链式思维重排方法(CoTRR),设计了列表级重排提示,使MLLM能直接参与候选图像的重排。该过程基于图像评估提示,评估每个候选图像与用户查询的匹配程度。通过让MLLM执行列表级推理,本方法支持全局比较、一致推理和可解释决策,这些对准确图像检索至关重要。为进一步实现结构化细粒度分析,我们引入查询分解提示,将原始查询拆分为多个语义成分。在五个数据集上的大量实验表明,CoTRR在文本到图像检索(TIR)、组合图像检索(CIR)和基于聊天的图像检索(Chat-IR)三个任务中均达到最先进性能。代码已开源。
原文摘要 · Abstract (English)
Image retrieval remains a fundamental yet challenging problem in computer vision. While recent advances in Multimodal Large Language Models (MLLMs) have demonstrated strong reasoning capabilities, existing methods typically employ them only for evaluation, without involving them directly in the ranking process. As a result, their rich multimodal reasoning abilities remain underutilized, leading to suboptimal performance. In this paper, we propose a novel Chain-of-Thought Re-Ranking (CoTRR) method to address this issue. Specifically, we design a listwise ranking prompt that enables MLLM to directly participate in re-ranking candidate images. This ranking process is grounded in an image evaluation prompt, which assesses how well each candidate aligns with users query. By allowing MLLM to perform listwise reasoning, our method supports global comparison, consistent reasoning, and interpretable decision-making - all of which are essential for accurate image retrieval. To enable structured and fine-grained analysis, we further introduce a query deconstruction prompt, which breaks down the original query into multiple semantic components. Extensive experiments on five datasets demonstrate the effectiveness of our CoTRR method, which achieves state-of-the-art performance across three image retrieval tasks, including text-to-image retrieval (TIR), composed image retrieval (CIR) and chat-based image retrieval (Chat-IR). Our code is available at https://github.com/freshfish15/CoTRR .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。