无需训练,通过双路检索与自反思提升零样本图像组合检索效果
WISER: Wider Search, Deeper Thinking, and Adaptive Fusion for Training-Free Zero-Shot Composed Image Retrieval
- 构建双向检索框架,同时生成编辑图文与图像进行并行搜索
- 在CIRCO上mAP@5提升45%,在CIRR上Recall@1提升57%超过以往方法
- 适合需要高精度零样本检索的视觉理解与跨模态应用
零样本组合图像检索(ZS-CIR)旨在给定包含参考图像和修改文本的多模态查询时,无需在标注三元组上训练即可检索目标图像。现有方法通常将多模态查询转换为单一模态:或作为编辑后的文本用于文本到图像检索(T2I),或作为编辑后的图像用于图像到图像检索(I2I)。但二者各有局限:T2I常丢失细粒度视觉细节,而I2I难以处理复杂语义修改。为有效利用二者互补优势,我们提出WISER,一种无需训练的统一框架,通过“检索-验证-精炼”流程融合T2I与I2I,并显式建模意图感知与不确定性感知。WISER首先通过生成编辑文本与图像实现更广范围搜索以扩大候选池;随后通过验证器评估置信度,对不确定结果触发精炼,并对可靠结果动态融合双路径结果;对于不确定结果,通过结构化自我反思生成精炼建议,引导下一轮深入思考。大量实验表明,WISER显著优于现有方法,在多个基准上实现相对提升:在CIRCO上mAP@5提高45%,在CIRR上Recall@1提高57%。尤其值得注意的是,其性能甚至超越许多依赖训练的方法,凸显其在多样化场景下的优越性与泛化能力。代码将发布于 https://github.com/Physicsmile/WISER。
原文摘要 · Abstract (English)
Zero-Shot Composed Image Retrieval (ZS-CIR) aims to retrieve target images given a multimodal query (comprising a reference image and a modification text), without training on annotated triplets. Existing methods typically convert the multimodal query into a single modality-either as an edited caption for Text-to-Image retrieval (T2I) or as an edited image for Image-to-Image retrieval (I2I). However, each paradigm has inherent limitations: T2I often loses fine-grained visual details, while I2I struggles with complex semantic modifications. To effectively leverage their complementary strengths under diverse query intents, we propose WISER, a training-free framework that unifies T2I and I2I via a "retrieve-verify-refine" pipeline, explicitly modeling intent awareness and uncertainty awareness. Specifically, WISER first performs Wider Search by generating both edited captions and images for parallel retrieval to broaden the candidate pool. Then, it conducts Adaptive Fusion with a verifier to assess retrieval confidence, triggering refinement for uncertain retrievals, and dynamically fusing the dual-path for reliable ones. For uncertain retrievals, WISER generates refinement suggestions through structured self-reflection to guide the next retrieval round toward Deeper Thinking. Extensive experiments demonstrate that WISER significantly outperforms previous methods across multiple benchmarks, achieving relative improvements of 45% on CIRCO (mAP@5) and 57% on CIRR (Recall@1) over existing training-free methods. Notably, it even surpasses many training-dependent methods, highlighting its superiority and generalization under diverse scenarios. Code will be released at https://github.com/Physicsmile/WISER.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。