arXiv:2507.12819cs.CV2025-07被引 3

不训练的零样本图像检索新方法,提升图文组合查询准确率

MCoT-RE: Multi-Faceted Chain-of-Thought and Re-Ranking for Training-Free Zero-Shot Composed Image Retrieval

  • 用多角度思维链引导模型兼顾文本修改与参考图上下文
  • 两阶段处理:先筛选候选图,再多粒度重排序提升精度
  • 适用于无训练数据场景,适合追求高精度零样本检索的研究者

组合图像检索(CIR)旨在通过包含参考图像和修改文本的复合查询,从图库中检索目标图像。现有基于预训练模型的免训练零样本方法虽成本低,但仍存在明显局限:顺序处理模态的方法易丢失信息,限制跨模态交互;而依赖多模态大语言模型(MLLM)的方法常仅关注文本指示的变化,未能充分利用参考图像的上下文视觉信息。为此,本文提出免训练零样本的多方面思维链与重排序框架(MCoT-RE)。该方法通过多方面思维链引导MLLM平衡显式修改与视觉上下文,生成两个不同侧重的描述:一个聚焦于修改内容,另一个融合完整图文上下文。前者用于初步候选图像筛选,后者结合参考图像进行多粒度重排序。该两阶段策略在遵循文本修改指令的同时,保留了参考图像的视觉特征,实现精准检索。大量实验表明,MCoT-RE在免训练方法中达到当前最优性能,在FashionIQ上Recall@10提升达6.24%,在CIRR上Recall@1提升达8.58%。

原文摘要 · Abstract (English)

Composed Image Retrieval (CIR) is the task of retrieving a target image from a gallery using a composed query consisting of a reference image and a modification text. Among various CIR approaches, training-free zero-shot methods based on pre-trained models are cost-effective but still face notable limitations. For example, sequential VLM-LLM pipelines process each modality independently, which often results in information loss and limits cross-modal interaction. In contrast, methods based on multimodal large language models (MLLMs) often focus exclusively on applying changes indicated by the text, without fully utilizing the contextual visual information from the reference image. To address these issues, we propose multi-faceted Chain-of-Thought with re-ranking (MCoT-RE), a training-free zero-shot CIR framework. MCoT-RE utilizes multi-faceted Chain-of-Thought to guide the MLLM to balance explicit modifications and contextual visual cues, generating two distinct captions: one focused on modification and the other integrating comprehensive visual-textual context. The first caption is used to filter candidate images. Subsequently, we combine these two captions and the reference image to perform multi-grained re-ranking. This two-stage approach facilitates precise retrieval by aligning with the textual modification instructions while preserving the visual context of the reference image. Through extensive experiments, MCoT-RE achieves state-of-the-art results among training-free methods, yielding improvements of up to 6.24% in Recall@10 on FashionIQ and 8.58% in Recall@1 on CIRR.

图像检索零样本多模态思维链

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。