arXiv:2502.20826cs.CVcs.IR2025-02ICCV被引 14

无需训练即可精准检索组合图像,靠分步推理与多尺度分析提升准确性。

CoTMR: Chain-of-Thought Multi-Scale Reasoning for Training-Free Zero-Shot Composed Image Retrieval

  • 用大视觉语言模型统一理解图文组合查询,避免模态转换误差。
  • 在四个基准上显著超越现有方法,最高提升17.3个百分点。
  • 适合需要可解释性检索的科研与工业场景,如医疗图像分析。

零样本组合图像检索(ZS-CIR)旨在不依赖训练样本的情况下,通过整合参考图像和修改文本组成的查询来检索目标图像。现有方法主要依赖图像描述模型与大语言模型(LLM)生成目标描述,但存在模态不兼容、视觉信息丢失及推理不足等问题。本文提出CoTMR,一种无需训练的ZS-CIR框架,引入链式思维(CoT)与多尺度推理机制。CoTMR采用大视觉语言模型(LVLM)实现对组合查询的统一理解与推理,避免使用图像描述模型进行模态转换。为增强推理可靠性,设计了CIRCoT,通过预定义子任务引导LVLM进行分步推理。针对现有方法仅关注全局推理的问题,引入多尺度推理,在物体级别精细判断关键元素是否存在。进一步设计多粒度评分(MGS)机制,将上述推理输出与候选图像的CLIP相似度融合,实现精确检索。大量实验表明,CoTMR在四个主流基准上均显著优于先前方法,最高提升达17.3个百分点,并具备良好可解释性。

原文摘要 · Abstract (English)

Zero-Shot Composed Image Retrieval (ZS-CIR) aims to retrieve target images by integrating information from a composed query (reference image and modification text) without training samples. Existing methods primarily combine caption models and large language models (LLMs) to generate target captions based on composed queries but face various issues such as incompatibility, visual information loss, and insufficient reasoning. In this work, we propose CoTMR, a training-free framework crafted for ZS-CIR with novel Chain-of-thought (CoT) and Multi-scale Reasoning. Instead of relying on caption models for modality transformation, CoTMR employs the Large Vision-Language Model (LVLM) to achieve unified understanding and reasoning for composed queries. To enhance the reasoning reliability, we devise CIRCoT, which guides the LVLM through a step-by-step inference process using predefined subtasks. Considering that existing approaches focus solely on global-level reasoning, our CoTMR incorporates multi-scale reasoning to achieve more comprehensive inference via fine-grained predictions about the presence or absence of key elements at the object scale. Further, we design a Multi-Grained Scoring (MGS) mechanism, which integrates CLIP similarity scores of the above reasoning outputs with candidate images to realize precise retrieval. Extensive experiments demonstrate that our CoTMR not only drastically outperforms previous methods across four prominent benchmarks but also offers appealing interpretability.

图像检索多尺度推理零样本大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。