直接组合图文生成查询表示，实现高效零样本图像检索
DiCE-CIR: Direct Composition Learning for Efficient Zero-Shot Composed Image Retrieval
- 直接拼接图像与文本生成查询向量，跳过投影和重编码步骤
- 在CIRCO上达到新最优性能，推理效率显著提升
- 适合需要快速响应的跨模态检索场景
零样本图像组合检索（ZS-CIR）旨在根据包含参考图像和编辑描述的多模态查询，从数据库中检索目标图像。现有方法依赖投影机制将图像映射到文本嵌入空间的伪词标记，需额外投影与重编码步骤，增加训练复杂度，降低效率，并引入训练-推理不一致问题。本文提出DiCE-CIR，一种直接组合学习方法，通过直接拼接参考图像与编辑文本生成组合查询表示。为实现可扩展训练，我们利用大语言模型从大规模图像-标题对中自动生成组合训练样本，无需人工标注三元组。基于这些样本，训练一个轻量级组合模块，优化目标包括：与目标对齐、编辑语义一致性转换以及检索判别力。在多个ZS-CIR基准上进行广泛实验，结果表明，DiCE-CIR在CIRCO上达到当前最优性能，在CIRR上表现具有竞争力，同时保持高计算效率。
原文摘要 · Abstract (English)
Zero-shot composed image retrieval (ZS-CIR) aims to retrieve a target image from a multimodal query consisting of a reference image and an edit text describing the desired modification. Recent ZS-CIR studies have relied on projection-based methods that map a reference image into pseudo-word tokens in the text embedding space. However, such methods require additional projection and re-encoding steps, increasing training complexity, reducing efficiency, and introducing a discrepancy between training and inference. In this paper, we propose DiCE-CIR, a direct composition learning method that predicts composed query representations by directly composing a reference image and an edit text. To enable scalable training without manually annotated triplets, we automatically construct compositional training samples from large-scale image-caption pairs using a large language model. Based on these samples, we train a lightweight composition module with objectives that promote alignment with the target, edit-consistent semantic transformation, and retrieval discriminability. We conduct extensive experiments on ZS-CIR benchmarks and show that DiCE-CIR achieves state-of-the-art performance on CIRCO and competitive performance on CIRR while maintaining high computational efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。