用提示方向向量动态调整图文检索,提升零样本组合图像搜索效果
PDV: Prompt Directional Vectors for Zero-shot Composed Image Retrieval
- 引入提示方向向量,让文本嵌入随提示动态变化
- 融合图文特征后,在多个数据集上准确率显著提升
- 无需训练、可直接插入现有方法,适合快速优化检索系统
零样本组合图像检索(ZS-CIR)允许用户仅通过参考图像和文本提示进行图像搜索,而无需依赖大规模配对数据训练的专用图文组合模型。然而,现有方法在使用组合文本嵌入时存在三大缺陷:查询嵌入静态不变、图像嵌入利用不足、图文融合效果不佳。为此,本文提出无需训练的「提示方向向量」(PDV),捕捉提示带来的语义变化。PDV实现三项改进:(1) 动态组合文本嵌入,通过缩放因子控制提示调整;(2) 将文本语义迁移至图像特征,生成组合图像嵌入;(3) 加权融合图文嵌入,平衡视觉与语义相似性。该方法作为即插即用模块,计算开销极小。大量实验表明,集成到主流ZS-CIR方法后,性能持续提升,尤其在生成精准组合嵌入的方法中表现更优。
原文摘要 · Abstract (English)
Zero-shot Composed Image Retrieval (ZS-CIR) enables image search using a reference image and a text prompt without requiring specialized text-image composition networks trained on large-scale paired data. However, current ZS-CIR approaches suffer from three critical limitations in their reliance on composed text embeddings: static query embedding representations, insufficient utilization of image embeddings, and suboptimal performance when fusing text and image embeddings. To address these challenges, we introduce the \textbf{Prompt Directional Vector (PDV)}, a simple yet effective training-free enhancement that captures semantic modifications induced by user prompts. PDV enables three key improvements: (1) Dynamic composed text embeddings where prompt adjustments are controllable via a scaling factor, (2) composed image embeddings through semantic transfer from text prompts to image features, and (3) weighted fusion of composed text and image embeddings that enhances retrieval by balancing visual and semantic similarity. Our approach serves as a plug-and-play enhancement for existing ZS-CIR methods with minimal computational overhead. Extensive experiments across multiple benchmarks demonstrate that PDV consistently improves retrieval performance when integrated with state-of-the-art ZS-CIR approaches, particularly for methods that generate accurate compositional embeddings. The code will be released upon publication.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。