arXiv:2505.19952cs.CVcs.IR2025-05被引 9

不用中间文本,直接学习图文组合检索关系。

Multimodal Reasoning Agent for Zero-Shot Composed Image Retrieval

  • 用多模态推理代理直接构建图像-文本-图像三元组
  • 在三个数据集上召回率提升7.5%至9.6%
  • 适合零样本图像组合检索场景

零样本组合图像检索(ZS-CIR)旨在给定由参考图像和修改文本组成的复合查询时,检索目标图像,且无需标注训练数据。现有方法通常利用大语言模型生成合成文本作为查询与目标图像之间的中间桥梁,再通过对比学习分别对齐查询-文本和图像-文本。然而,这种依赖中间文本的机制会导致误差传播,因查询到文本、文本到图像的映射不准确而累积,最终降低检索性能。为此,本文提出一种新框架——多模态推理代理(MRA),用于零样本组合图像检索。MRA 通过仅使用未标注图像数据,直接构建三元组 <参考图像, 修改文本, 目标图像>,消除对文本中介的依赖。模型在这些合成三元组上训练,直接学习复合查询与候选图像间的关联关系。在三个标准 CIR 基准上的大量实验表明,本方法效果显著:在 FashionIQ 数据集上,平均 R@10 提升至少 7.5%;在 CIRR 上,R@1 提升 9.6%;在 CIRCO 上,mAP@5 提升 9.5%。

原文摘要 · Abstract (English)

Zero-Shot Composed Image Retrieval (ZS-CIR) aims to retrieve target images given a compositional query, consisting of a reference image and a modifying text-without relying on annotated training data. Existing approaches often generate a synthetic target text using large language models (LLMs) to serve as an intermediate anchor between the compositional query and the target image. Models are then trained to align the compositional query with the generated text, and separately align images with their corresponding texts using contrastive learning. However, this reliance on intermediate text introduces error propagation, as inaccuracies in query-to-text and text-to-image mappings accumulate, ultimately degrading retrieval performance. To address these problems, we propose a novel framework by employing a Multimodal Reasoning Agent (MRA) for ZS-CIR. MRA eliminates the dependence on textual intermediaries by directly constructing triplets, <reference image, modification text, target image>, using only unlabeled image data. By training on these synthetic triplets, our model learns to capture the relationships between compositional queries and candidate images directly. Extensive experiments on three standard CIR benchmarks demonstrate the effectiveness of our approach. On the FashionIQ dataset, our method improves Average R@10 by at least 7.5\% over existing baselines; on CIRR, it boosts R@1 by 9.6\%; and on CIRCO, it increases mAP@5 by 9.5\%.

图像检索多模态零样本推理代理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。