用单阶段反思链思维,精准实现零样本图像组合检索。
Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrieval
- 单阶段推理保留视觉细节,避免两阶段信息丢失。
- 在多个任务上提升1.80%至6.44%,达新SOTA。
- 适合需要高精度视觉语言理解的开发者与研究者。
组合图像检索(CIR)旨在根据参考图像和用户指定的文本修改,检索出与目标意图更贴合的图像,从而更精确地捕捉用户需求。现有无训练零样本CIR(ZS-CIR)方法通常采用两阶段流程:先生成参考图像的描述,再利用大语言模型进行推理以获取目标描述。然而,这类方法常因遗漏关键视觉细节且推理能力有限,导致检索性能不佳。为此,我们提出一种新型无训练单阶段方法——OSrCIR,通过多模态大语言模型在单一推理过程中保留核心视觉信息,消除两阶段方法中的信息损失。我们的反思链思维框架进一步通过将操作意图与参考图像的上下文线索对齐,提升解释准确性。OSrCIR在多个任务上相较现有方法提升1.80%至6.44%,刷新了零样本图像检索的性能上限,显著增强了其在视觉-语言应用中的实用性。代码将发布于 https://github.com/Pter61/osrcir2024/。
原文摘要 · Abstract (English)
Composed Image Retrieval (CIR) aims to retrieve target images that closely resemble a reference image while integrating user-specified textual modifications, thereby capturing user intent more precisely. Existing training-free zero-shot CIR (ZS-CIR) methods often employ a two-stage process: they first generate a caption for the reference image and then use Large Language Models for reasoning to obtain a target description. However, these methods suffer from missing critical visual details and limited reasoning capabilities, leading to suboptimal retrieval performance. To address these challenges, we propose a novel, training-free one-stage method, One-Stage Reflective Chain-of-Thought Reasoning for ZS-CIR (OSrCIR), which employs Multimodal Large Language Models to retain essential visual information in a single-stage reasoning process, eliminating the information loss seen in two-stage methods. Our Reflective Chain-of-Thought framework further improves interpretative accuracy by aligning manipulation intent with contextual cues from reference images. OSrCIR achieves performance gains of 1.80% to 6.44% over existing training-free methods across multiple tasks, setting new state-of-the-art results in ZS-CIR and enhancing its utility in vision-language applications. Our code will be available at https://github.com/Pter61/osrcir2024/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。