首个统一零样本视觉检索框架,一次解决三类组合式图文视频检索问题。
UniCVR: From Alignment to Reranking for Unified Zero-Shot Composed Visual Retrieval

- 用多模态大模型理解复杂查询,结合预训练视觉语言模型实现结构化检索。
- 在5个基准上达到顶尖性能,350万样本训练提升跨模态对齐效果。
- 适合需要通用检索能力的研究者,尤其关注零样本场景的应用落地。
组合式图像检索、多轮组合图像检索和组合式视频检索均遵循同一范式:通过修改文本与参考视觉内容组合来检索目标。尽管结构一致,三类任务长期独立研究,尚无统一框架,更无零样本解决方案。本文提出UniCVR,首个统一的零样本组合视觉检索框架,无需特定任务标注数据即可同时处理三项任务。该方法分两阶段:第一阶段在约350万样本的多源数据集上,通过对比学习训练多模态大语言模型(MLLM)作为组合查询编码器,桥接MLLM与冻结的视觉语言预训练(VLP)图库编码器间的异构嵌入空间,并采用基于聚类的难负样本采样策略增强对比监督;第二阶段引入MLLM引导的双层重排序机制,对少数顶级候选进行自适应预算子集评分,并利用重排序结果实施双层再评分,以极低计算开销获得更精准最终排名。在覆盖三项任务的五个基准上的广泛实验表明,UniCVR性能达到前沿水平,验证其有效性和泛化能力。
原文摘要 · Abstract (English)
Composed image retrieval, multi-turn composed image retrieval, and composed video retrieval all share a common paradigm: composing the reference visual with modification text to retrieve the desired target. Despite this shared structure, the three tasks have been studied in isolation, with no prior work proposing a unified framework, let alone a zero-shot solution. In this paper, we propose UniCVR, the first unified zero-shot composed visual retrieval framework that jointly addresses all three tasks without any task-specific human-annotated data. UniCVR strategically combines two complementary strengths: Multimodal Large Language Models (MLLMs) for compositional query understanding and Vision-Language Pre-trained (VLP) models for structured visual retrieval. Concretely, UniCVR operates in two stages. In Stage~I, we train the MLLM as a compositional query embedder via contrastive learning on a curated multi-source dataset of approximately 3.5M samples, bridging the heterogeneous embedding spaces between the MLLM and the frozen VLP gallery encoder. A cluster-based hard negative sampling strategy is proposed to strengthen contrastive supervision. In Stage~II, we introduce an MLLM-guided dual-level reranking mechanism that applies adaptive budgeted subset scoring to a small number of top-ranked candidates, and then exploits the resulting relevance signals through a dual-level re-scoring scheme, producing more accurate final rankings with minimal computational overhead. Extensive experiments across five benchmarks covering all three tasks demonstrate that UniCVR achieves cutting-edge performance, validating its effectiveness and generalizability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。