arXiv:2608.23102cs.CVcs.IR2026-08

无需训练,用生成模型实现图像组合检索的跨模态融合。

Training-Free Pseudo-Fusion for Composed Image Retrieval with Diffusion Models and Multimodal Large Language Models

论文配图:Training-Free Pseudo-Fusion for Composed Image Retrieval with Diffusion Models and Multimodal Large Language Models
图 1 · 摘自论文原文
  • 用生成式转换将多模态查询转为单模态检索任务。
  • 在标准数据集上性能媲美或超越现有方法。
  • 适合希望快速部署零样本检索系统的开发者。

组合图像检索(CIR)是一种新兴的内容驱动图像检索范式,允许用户通过参考图像与辅助模态(通常是文本)组合来提出查询。该方法支持细粒度搜索:目标图像与用户提供的图像共享结构元素,同时融入辅助文本指定的修改。传统CIR方法依赖多模态融合将视觉与文本特征结合为联合查询嵌入,需训练对齐模块。本文提出无训练框架PeFuse(伪融合),利用预训练扩散模型与多模态大语言模型,通过生成式转换桥接模态。引入单向与双向转换策略,将CIR重构为四个单模态检索问题。这些方法将CIR重新定义为同模态或跨模态单查询检索任务,避免了专用任务训练。在标准基准上的大量实验表明,将CIR转为文本到图像检索更有效,性能达到或优于当前最优方法,且得益于可替换的转换组件保持高灵活性。结果凸显了伪融合范式的有效性,适用于零样本CIR。代码已公开于:https://github.com/StevenXuf/PeFuse4CIR。

原文摘要 · Abstract (English)

Composed Image Retrieval (CIR) is an emerging paradigm in content-based image retrieval that enables users to formulate compositional queries by combining a reference image with an auxiliary modality, usually text-based. This approach supports fine-grained search where the target image shares structural elements with the user-provided image while incorporating the modifications specified by the auxiliary text. Conventional CIR methods rely on multimodal fusion to combine visual and textual features into a joint query embedding, which requires training modules that align composed queries with the targets. In this work, we propose PeFuse (for pseudo-fusion), a training-free framework that leverages pretrained Diffusion Models and Multimodal Large Language Models to bridge modalities via generative conversion. We introduce two novel strategies: uni-directional and bi-directional conversion, which convert CIR into four single-modality retrieval problems. These methods reformulate CIR as either intra-modal or cross-modal single-query retrieval tasks, bypassing the need for dedicated task-specific training. Extensive experiments on standard benchmarks demonstrate that converting CIR into text-to-image retrieval tasks is more effective than alternative conversion strategies, achieving competitive or superior performance compared with state-of-the-art methods, while maintaining high flexibility thanks to replaceable components of the conversion pipeline. These results highlight the effectiveness of the pseudo-fusion paradigm for zero-shot CIR. Our code is publicly available at: https://github.com/StevenXuf/PeFuse4CIR.

图像检索生成模型零样本多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。