解决零样本图像检索中的语义偏差问题,提升生成描述准确性
SDR-CIR: Semantic Debias Retrieval Framework for Training-Free Zero-Shot Composed Image Retrieval
- 用选择性思维链筛选修改文本相关视觉内容,减少噪声
- 分两步去偏:锚定增强有用语义,去偏惩罚冗余参考特征
- 无需训练,在三个基准上达到领先效果,适合高效检索场景
组成式图像检索(CIR)旨在根据参考图像和修改文本的组合查询,从数据库中检索目标图像。现有无训练零样本方法常利用多模态大语言模型(MLLM)与思维链(CoT)生成目标图像描述进行检索,但由于零样本检索的模糊匹配特性,生成描述易产生相对于目标图像的语义偏差。本文提出SDR-CIR,一种基于CoT推理的无训练语义去偏排序方法。首先,选择性思维链引导MLLM在理解图像时提取与修改文本相关的视觉内容,从而在源头减少视觉噪声。随后引入两步语义去偏排序机制:锚定步骤将参考图像特征与目标描述特征融合,强化有效语义并补充遗漏线索;去偏步骤显式建模参考图像对描述的视觉语义贡献,并将其作为惩罚项融入相似度分数。通过补充遗漏线索并抑制冗余,SDR-CIR有效缓解语义偏差,提升检索性能。在三个标准CIR基准上的实验表明,SDR-CIR在单阶段方法中达到最新水平,同时保持高效率。代码已公开于https://github.com/suny105/SDR-CIR。
原文摘要 · Abstract (English)
Composed Image Retrieval (CIR) aims to retrieve a target image from a query composed of a reference image and modification text. Recent training-free zero-shot methods often employ Multimodal Large Language Models (MLLMs) with Chain-of-Thought (CoT) to compose a target image description for retrieval. However, due to the fuzzy matching nature of ZS-CIR, the generated description is prone to semantic bias relative to the target image. We propose SDR-CIR, a training-free Semantic Debias Ranking method based on CoT reasoning. First, Selective CoT guides the MLLM to extract visual content relevant to the modification text during image understanding, thereby reducing visual noise at the source. We then introduce a Semantic Debias Ranking with two steps, Anchor and Debias, to mitigate semantic bias. In the Anchor step, we fuse reference image features with target description features to reinforce useful semantics and supplement omitted cues. In the Debias step, we explicitly model the visual semantic contribution of the reference image to the description and incorporate it into the similarity score as a penalty term. By supplementing omitted cues while suppressing redundancy, SDR-CIR mitigates semantic bias and improves retrieval performance. Experiments on three standard CIR benchmarks show that SDR-CIR achieves state-of-the-art results among one-stage methods while maintaining high efficiency. The code is publicly available at https://github.com/suny105/SDR-CIR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。