用多模态推理精准选图,提升组合图像检索准确率
MCoT-MVS: Multi-level Vision Selection by Multi-modal Chain-of-Thought Reasoning for Composed Image Retrieval
- 通过多模态思维链分析文本与图像,生成视觉选择线索
- 在CIRR和FashionIQ上超越现有方法,最高提升12.3%
- 适合需要精准理解图文修改意图的图像检索场景
组合图像检索(CIR)旨在根据参考图像和修改后的文本检索目标图像。然而,现有方法常因无法从参考图像中提取正确语义线索而受无关视觉噪声干扰。本文提出一种基于多模态思维链推理的多层级视觉选择方法(MCoT-MVS),利用多模态大模型(MLLM)对多模态输入进行推理,生成保留、移除及目标推断文本。这些文本线索引导两个视觉注意力模块,分别提取参考图像的片段级与实例级语义特征。最后,设计加权分层融合模块,将多粒度视觉特征与修改后文本及想象中的目标描述对齐至统一嵌入空间。在CIRR与FashionIQ两个基准上的大量实验表明,该方法持续优于现有方法,达到新最佳性能。代码与训练模型已公开。
原文摘要 · Abstract (English)
Composed Image Retrieval (CIR) aims to retrieve target images based on a reference image and modified texts. However, existing methods often struggle to extract the correct semantic cues from the reference image that best reflect the user's intent under textual modification prompts, resulting in interference from irrelevant visual noise. In this paper, we propose a novel Multi-level Vision Selection by Multi-modal Chain-of-Thought Reasoning (MCoT-MVS) for CIR, integrating attention-aware multi-level vision features guided by reasoning cues from a multi-modal large language model (MLLM). Specifically, we leverage an MLLM to perform chain-of-thought reasoning on the multimodal composed input, generating the retained, removed, and target-inferred texts. These textual cues subsequently guide two reference visual attention selection modules to selectively extract discriminative patch-level and instance-level semantics from the reference image. Finally, to effectively fuse these multi-granular visual cues with the modified text and the imagined target description, we design a weighted hierarchical combination module to align the composed query with target images in a unified embedding space. Extensive experiments on two CIR benchmarks, namely CIRR and FashionIQ, demonstrate that our approach consistently outperforms existing methods and achieves new state-of-the-art performance. Code and trained models are publicly released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。