arXiv:2509.26012cs.CV2025-09被引 2

通过两阶段框架提升零样本图像检索精度,有效过滤无关细节并精准匹配文本修改。

SETR: A Two-Stage Semantic-Enhanced Framework for Zero-Shot Composed Image Retrieval

  • 先用交集策略聚焦参考图与文本共有的语义,去除干扰信息
  • 再用低秩适配的多模态大模型做细粒度是否匹配判断,提升准确性
  • 在多个数据集上超越现有方法,尤其在CIRR上提升15.15点

零样本组合图像检索(ZS-CIR)旨在给定参考图像和相对文本时检索目标图像,无需依赖昂贵的三元组标注。现有基于CLIP的方法面临两大挑战:(1) 联合特征融合无差别聚合所有视觉线索,引入无关背景信息,稀释目标修改;(2) CLIP嵌入的全局余弦相似度无法解析细粒度语义关系。为此,我们提出语义增强型两阶段检索框架SETR。在粗检索阶段,引入交集驱动策略,仅保留参考图与相对文本之间的重叠语义,过滤联合融合固有的干扰项,生成更干净、高精度的候选集。在细粒度重排序阶段,采用低秩适配(LoRA)微调预训练多模态大模型进行二分类语义相关性判断(“是/否”),突破CLIP的全局匹配限制,显式验证关系与属性一致性。两个阶段形成互补:粗检索以高召回率缩小候选范围,重排序确保与文本细微修改精确对齐。在CIRR、Fashion-IQ和CIRCO上的实验表明,SETR达到新最优性能,在CIRR上Recall@1最高提升15.15点。结果确立了两阶段推理作为鲁棒且可迁移的ZS-CIR通用范式。

原文摘要 · Abstract (English)

Zero-shot Composed Image Retrieval (ZS-CIR) aims to retrieve a target image given a reference image and a relative text, without relying on costly triplet annotations. Existing CLIP-based methods face two core challenges: (1) union-based feature fusion indiscriminately aggregates all visual cues, carrying over irrelevant background details that dilute the intended modification, and (2) global cosine similarity from CLIP embeddings lacks the ability to resolve fine-grained semantic relations. To address these issues, we propose SETR (Semantic-enhanced Two-Stage Retrieval). In the coarse retrieval stage, SETR introduces an intersection-driven strategy that retains only the overlapping semantics between the reference image and relative text, thereby filtering out distractors inherent to union-based fusion and producing a cleaner, high-precision candidate set. In the fine-grained re-ranking stage, we adapt a pretrained multimodal LLM with Low-Rank Adaptation to conduct binary semantic relevance judgments ("Yes/No"), which goes beyond CLIP's global feature matching by explicitly verifying relational and attribute-level consistency. Together, these two stages form a complementary pipeline: coarse retrieval narrows the candidate pool with high recall, while re-ranking ensures precise alignment with nuanced textual modifications. Experiments on CIRR, Fashion-IQ, and CIRCO show that SETR achieves new state-of-the-art performance, improving Recall@1 on CIRR by up to 15.15 points. Our results establish two-stage reasoning as a general paradigm for robust and portable ZS-CIR.

图像检索零样本多模态两阶段

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。