让图像搜索能像对话一样逐步优化结果，支持多轮交互。
CoCo-IR: Contextual Composed Image Retrieval
- 用大模型理解对话历史，动态生成可演化图像嵌入
- 在4轮对话中达44.1 R@1，远超现有方法的28.2
- 自动生成高质量训练数据，无需人工标注
当前基于指令的图像检索系统虽强大，但仅支持单轮交互，难以应对复杂真实的视觉搜索场景。为此，我们提出新任务——上下文组合图像检索（CoCo-IR），支持用户通过多轮交互逐步精炼搜索结果。我们设计基于大模态模型（LMM）的新模型，作为上下文感知推理器，利用完整对话历史生成可变换的图像嵌入（TIE），实现跨轮次的动态演化。为避免昂贵的人工标注，我们构建全自动化、可扩展的数据引擎，借助LMM生成高质量上下文检索数据，并采用模型引导验证挖掘挑战性难负例。大量实验表明，本方法达到新最优性能：在挑战性单轮基准CIRCO上取得39.4 mAP@5；在新提出的CoCo-IR基准上，4轮对话下仍保持44.1 R@1，显著优于现有方法（28.2 R@1），后者无法处理多轮上下文。
原文摘要 · Abstract (English)
Current instruction-based image retrieval systems are powerful but limited to single-turn interactions, failing to capture the iterative nature of complex, real-world visual searches. To overcome this limitation, we introduce Contextual Composed Image Retrieval (CoCo-IR), a novel task that enables users to progressively refine search results through interactions. We address this new task by proposing a new model based on a Large Multimodal Model (LMM) that functions as a context-aware reasoner for CoCo-IR. Our model interprets the entire interaction history to generate Transformable Image Embeddings (TIE) that evolve across turns. To fuel the model training without expensive human annotations, we develop a fully autonomous, scalable data engine that leverages LMMs to generate high-quality contextual retrieval data, and uses model-guided verification to mine challenging hard negatives. Extensive experiments demonstrate that our approach establishes new state-of-the-art performance: We achieve 39.4 mAP@5 on the challenging single-turn benchmark CIRCO; furthermore, on our new CoCo-IR benchmark, our model maintains robust performance with 44.1 R@1 on 4-turn dialogues, dramatically outperforming existing methods (28.2 4-turn R@1) that fail to handle multi-turn context. Project page: https://CoCo-IR.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。