arXiv:2605.21261cs.CV2026-05被引 1

无需训练,通过语义迁移实现零样本图像检索,更准更灵活。

STiTch: Semantic Transition and Transportation in Collaboration for Training-Free Zero-Shot Composed Image Retrieval

论文配图:STiTch: Semantic Transition and Transportation in Collaboration for Training-Free Zero-Shot Composed Image Retrieval
图 1 · 摘自论文原文
  • 用嵌入空间的过渡向量优化大模型生成的描述,减少无关细节干扰。
  • 将检索任务转为集合对集合匹配,提升多样组合的识别能力。
  • 适合需要快速适配新场景的零样本图像检索应用。

免训练的零样本组合图像检索模型因具备良好的泛化性和灵活性而受到越来越多关注。当前基于大语言模型(LLM)的方法通过挖掘其组合生成能力来生成目标描述。然而,这类方法存在两个问题:1)生成描述时容易引入参考图像中未提及的额外特征,源于图像与文本修改之间的语义鸿沟;2)检索阶段的点对点对齐无法捕捉多样化组合。为此,我们提出一种新型的语义过渡与协同传输框架(STiTch),用于免训练零样本组合图像检索。具体地,给定由大语言模型推断出的组合描述,我们在嵌入空间中引入过渡向量,使其更贴近目标图像。结合用户指令,优化后的描述聚焦核心修改意图,有效过滤冗余噪声。此外,为探索检索阶段的多样性对齐,我们将描述与图像建模为离散分布,并将检索任务重构为集合对集合对齐。最后,设计双向运输距离以捕捉跨模态细粒度对齐并计算检索得分。大量实验表明,该方法在多种组合图像检索任务中具有通用性、有效性与优越性。

原文摘要 · Abstract (English)

Training-free zero-shot composed image retrieval models are recently gaining increasing research interest due to their generalizability and flexibility in unseen multimodal retrieval. Recent LLM-based advances focus on generating the expected target caption by exploring the compositional ability behind the LLMs. Although efficient, we find that 1) the generated captions tend to introduce unexpected features from the reference image due to the semantic gap between the input image and text modification, where the image contains much more details than the text; 2) the point-to-point alignment during the retrieval stage fails to capture diverse compositions. To address these challenges, we introduce a novel Semantic Transition and Transportation in collaboration framework for training-free zero-shot CIR tasks. Specifically, given the composed caption inferred by an LLM, we aim to refine it through a transition vector in the embedding space and make it closer to the target image. Combining LLMs with user instruction, the refined caption concentrates more on the core modification intent and thus filters out unnecessary noise. Moreover, to explore diverse alignment during the retrieval stage, we model the caption and image as discrete distributions and reformulate the retrieval task as a set-to-set alignment task. Finally, a bidirectional transportation distance is developed to consider fine-grained alignments across modalities and calculate the retrieval score. Extensive experiments demonstrate that our method can be general, effective, and beneficial for many CIR tasks.

零样本检索图像检索大模型组合生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。