解耦图像检索中的终点与语义变化学习,提升零样本组合检索效果。
Decoupling Endpoint and Semantic Transition Learning for Zero-Shot Composed Image Retrieval

- 分离终点对齐与语义过渡学习,避免训练冲突。
- 在多个数据集上显著超越现有投影方法,性能更优。
- 适合追求高效无大模型依赖的图像检索研究者。
零样本组合图像检索(ZS-CIR)从参考图像和文本修改中检索目标图像,无需人工标注的三元组。基于投影的方法因推理时不依赖大语言模型且轻量而受青睐,但在复杂语义修改下表现仍逊于基于LLM的方法。这反映了投影式ZS-CIR中的语义过渡瓶颈:终点匹配使编辑文本仅作为目标属性提示,而非作为源条件下的语义转变依据。我们进一步发现,在同一文本适配器中加入语义过渡监督,会导致终点对齐与语义过渡对齐之间的冲突。为解决此问题,DeCIR采用解耦策略:从图像-标题对构建正向/反向编辑对,分别训练低秩文本适配器分支以实现终点对齐与语义过渡对齐,并通过低秩方向融合(LRDM)合并为可部署适配器。在CIRR、CIRCO、FashionIQ和GeneCIS上的大量实验表明,DeCIR在不增加推理复杂度的前提下,持续提升投影式ZS-CIR性能。
原文摘要 · Abstract (English)
Zero-shot composed image retrieval (ZS-CIR) retrieves a target image from a reference image and a text modification without human-annotated CIR triplets. Projection-based ZS-CIR methods are attractive because they do not rely on LLMs at inference and remain lightweight, but they often underperform LLM-based approaches on complex semantic modifications. This gap reflects a semantic transition bottleneck in projection-based ZS-CIR: endpoint-level matching can let the edit text act as a target-side attribute cue rather than grounding it as a source-conditioned semantic transition. We further show that adding semantic transition supervision to the same text adapter creates an endpoint--transition conflict between endpoint alignment and semantic transition alignment. To address this conflict, DeCIR decouples endpoint and transition learning. It constructs paired forward/reverse edit tuples from image-caption pairs, trains separate low-rank text adapter branches for endpoint alignment and semantic transition alignment, and merges them with Low-Rank Directional Merge (LRDM) into one deployable adapter. Extensive experiments on CIRR, CIRCO, FashionIQ, and GeneCIS demonstrate that DeCIR consistently improves projection-based ZS-CIR without increasing inference complexity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。