通过融合图文特征提升零样本图像检索准确率，最高达50.4% Recall@10。
Zero Shot Composed Image Retrieval
- 用轻量Q-Former融合双模态特征，实现更精准的图文对齐。
- 在FashionIQ上平均Recall@50提升至67.6%，衬衫类达45.6%。
- 适合需要高精度零样本图像检索的研究与应用者。
组合图像检索（CIR）允许用户通过细粒度文本编辑（如“把裙子变成蓝色”或“去掉条纹”）从参考图中定位目标图像。零样本CIR采用独立预训练的视觉-语言编码器，仅在FashionIQ基准上达到20-25% Recall@10。本文通过微调BLIP-2，并引入轻量级Q-Former融合视觉与文本特征，生成单一嵌入表示，使Recall@10提升至45.6%（衬衫）、40.1%（连衣裙）、50.4%（上衣-T恤），平均Recall@50达67.6%。同时评估了基于偏好优化的Retrieval-DPO方法，尽管调整了缩放因子、索引和采样策略，其Recall@10仍仅为0.02%，远低于零样本与提示微调基线。原因包括：缺乏联合图文融合、使用与top-K指标不匹配的边际目标、依赖低质量负例，以及冻结视觉与Transformer层。结果表明，有效的基于偏好的CIR需具备真实多模态融合、排名感知目标及精心筛选的负例。
原文摘要 · Abstract (English)
Composed image retrieval (CIR) allows a user to locate a target image by applying a fine-grained textual edit (e.g., ``turn the dress blue'' or ``remove stripes'') to a reference image. Zero-shot CIR, which embeds the image and the text with separate pretrained vision-language encoders, reaches only 20-25\% Recall@10 on the FashionIQ benchmark. We improve this by fine-tuning BLIP-2 with a lightweight Q-Former that fuses visual and textual features into a single embedding, raising Recall@10 to 45.6\% (shirt), 40.1\% (dress), and 50.4\% (top-tee) and increasing the average Recall@50 to 67.6\%. We also examine Retrieval-DPO, which fine-tunes CLIP's text encoder with a Direct Preference Optimization loss applied to FAISS-mined hard negatives. Despite extensive tuning of the scaling factor, index, and sampling strategy, Retrieval-DPO attains only 0.02\% Recall@10 -- far below zero-shot and prompt-tuned baselines -- because it (i) lacks joint image-text fusion, (ii) uses a margin objective misaligned with top-$K$ metrics, (iii) relies on low-quality negatives, and (iv) keeps the vision and Transformer layers frozen. Our results show that effective preference-based CIR requires genuine multimodal fusion, ranking-aware objectives, and carefully curated negatives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。