用两阶段框架实现零样本图像组合检索,无需大量标注数据
From Mapping to Composing: A Two-Stage Framework for Zero-shot Composed Image Retrieval
- 先映射后组合:第一阶段增强图像到伪词的表征能力
- 仅需少量合成数据,第二阶段就可精准提取组合语义
- 对低质量合成数据也有效,适合数据稀缺场景
组合图像检索(CIR)是一项挑战性的多模态任务,旨在根据参考图像和修改文本检索目标图像。由于标注CIR三元组数据成本高,零样本(ZS)CIR成为有前景的替代方案。现有方法多采用基于投影的方法,将图像映射为单一伪词标记,但面临三个关键问题:(1)伪词标记表征能力不足;(2)训练与推理阶段不一致;(3)依赖大规模合成数据。为此,我们提出一个两阶段框架,从映射到组合完成训练。第一阶段通过引入视觉语义注入模块和软文本对齐目标,增强图像到伪词标记的学习,使标记能捕捉更丰富、细粒度的图像信息。第二阶段使用少量合成三元组数据优化文本编码器,使其通过结合伪词标记与修改文本,有效提取组合语义以实现准确的目标图像检索。第一阶段建立的强大视觉-伪词映射为第二阶段提供坚实基础,使方法兼容高质量与低质量合成数据,并在仅使用少量合成数据的情况下取得显著性能提升。在三个公开数据集上的大量实验表明,该方法优于现有主流方法。
原文摘要 · Abstract (English)
Composed Image Retrieval (CIR) is a challenging multimodal task that retrieves a target image based on a reference image and accompanying modification text. Due to the high cost of annotating CIR triplet datasets, zero-shot (ZS) CIR has gained traction as a promising alternative. Existing studies mainly focus on projection-based methods, which map an image to a single pseudo-word token. However, these methods face three critical challenges: (1) insufficient pseudo-word token representation capacity, (2) discrepancies between training and inference phases, and (3) reliance on large-scale synthetic data. To address these issues, we propose a two-stage framework where the training is accomplished from mapping to composing. In the first stage, we enhance image-to-pseudo-word token learning by introducing a visual semantic injection module and a soft text alignment objective, enabling the token to capture richer and fine-grained image information. In the second stage, we optimize the text encoder using a small amount of synthetic triplet data, enabling it to effectively extract compositional semantics by combining pseudo-word tokens with modification text for accurate target image retrieval. The strong visual-to-pseudo mapping established in the first stage provides a solid foundation for the second stage, making our approach compatible with both high- and low-quality synthetic data, and capable of achieving significant performance gains with only a small amount of synthetic data. Extensive experiments were conducted on three public datasets, achieving superior performance compared to existing approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。