无需训练即可实现跨模态图像检索，仅靠加权融合提升精准度。
Training-free Zero-shot Composed Image Retrieval via Weighted Modality Fusion and Similarity
- 用加权平均融合图像与文本特征生成查询表示
- 在FashionIQ和CIRR数据集上达到领先效果
- 适合追求快速部署的视觉搜索应用
组合图像检索（CIR）通过将参考图像与修改文本结合来表达查询，显著提升了用户意图的理解能力。然而，传统监督式训练需大量人工标注的三元组数据（参考图像、文本修改项、目标图像），成本高昂。现有零样本方法虽免去下游数据训练，但仍需大规模图像预训练。本文提出一种无需训练的零样本组合图像检索方法——WeiMoCIR，其核心思想是：通过简单加权平均可有效融合图像与文本模态。该方法直接基于参考图像和文本修改项构建查询表示，并利用多模态大模型为数据库图像生成文本描述，再以加权平均方式将文本描述与图像信息结合用于相似性计算。实验在FashionIQ和CIRR数据集上验证了其有效性，方法简洁易实现。代码已开源。
原文摘要 · Abstract (English)
Composed image retrieval (CIR), which formulates the query as a combination of a reference image and modified text, has emerged as a new form of image search due to its enhanced ability to capture user intent. However, training a CIR model in a supervised manner typically requires labor-intensive collection of (reference image, text modifier, target image) triplets. While existing zero-shot CIR (ZS-CIR) methods eliminate the need for training on specific downstream datasets, they still require additional pretraining on large-scale image datasets. In this paper, we introduce a training-free approach for ZS-CIR. Our approach, Weighted Modality fusion and similarity for CIR (WeiMoCIR), operates under the assumption that image and text modalities can be effectively combined using a simple weighted average. This allows the query representation to be constructed directly from the reference image and text modifier. To further enhance retrieval performance, we employ multimodal large language models (MLLMs) to generate image captions for the database images and incorporate these textual captions into the similarity computation by combining them with image information using a weighted average. Our approach is simple, easy to implement, and its effectiveness is validated through experiments on the FashionIQ and CIRR datasets. Code is available at https://github.com/whats2000/WeiMoCIR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。