让图像检索过程可解释,通过思维链逐步推理找到目标图。
CIR-CoT: Towards Interpretable Composed Image Retrieval via End-to-End Chain-of-Thought Reasoning
- 用思维链分步推理视觉与文本信息,提升跨模态理解。
- 在时尚和通用数据集上表现优于现有方法,跨域泛化能力强。
- 首次构建结构化推理标注数据,适合需要透明决策的场景。
组合图像检索(CIR)旨在根据参考图像和修改文本找出目标图像,其核心挑战在于统一处理视觉与语义模态的推理。尽管基于视觉-语言模型(如CLIP)和多模态大语言模型(如Qwen-VL)的方法已取得进展,但大多为“黑箱”模式,难以解释检索逻辑,也难以遵循复杂细粒度指令。为此,我们提出CIR-CoT,首个面向检索任务、端到端的多模态大语言模型,集成显式思维链(CoT)推理机制。通过强制模型先生成可解释的推理链条,增强对关键跨模态交互的捕捉能力,从而实现更精准的检索并公开决策过程。由于现有数据集(如FashionIQ、CIRR)缺乏推理标注,我们采用三阶段流程(描述、推理、结论)构建结构化CoT注释数据。模型经微调后输出该结构化结果,并将最终检索意图编码为专用嵌入。大量实验表明,CIR-CoT在域内数据集(FashionIQ、CIRR)上表现优异,在域外数据集CIRCO上展现出显著泛化能力,为更高效、可信的检索系统开辟新路径。
原文摘要 · Abstract (English)
Composed Image Retrieval (CIR), which aims to find a target image from a reference image and a modification text, presents the core challenge of performing unified reasoning across visual and semantic modalities. While current approaches based on Vision-Language Models (VLMs, e.g., CLIP) and more recent Multimodal Large Language Models (MLLMs, e.g., Qwen-VL) have shown progress, they predominantly function as ``black boxes." This inherent opacity not only prevents users from understanding the retrieval rationale but also restricts the models' ability to follow complex, fine-grained instructions. To overcome these limitations, we introduce CIR-CoT, the first end-to-end retrieval-oriented MLLM designed to integrate explicit Chain-of-Thought (CoT) reasoning. By compelling the model to first generate an interpretable reasoning chain, CIR-CoT enhances its ability to capture crucial cross-modal interactions, leading to more accurate retrieval while making its decision process transparent. Since existing datasets like FashionIQ and CIRR lack the necessary reasoning data, a key contribution of our work is the creation of structured CoT annotations using a three-stage process involving a caption, reasoning, and conclusion. Our model is then fine-tuned to produce this structured output before encoding its final retrieval intent into a dedicated embedding. Comprehensive experiments show that CIR-CoT achieves highly competitive performance on in-domain datasets (FashionIQ, CIRR) and demonstrates remarkable generalization on the out-of-domain CIRCO dataset, establishing a new path toward more effective and trustworthy retrieval systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。