让AI精准找到与查询图中同一物体匹配的场景,突破传统检索局限。
IDMR: Towards Instance-Driven Precise Visual Correspondence in Multimodal Retrieval
- 基于多模态大模型,实现跨场景的实例级图像匹配。
- 在55.7万合成样本上训练,零样本测试性能超越现有方法。
- 适合需要细粒度视觉对应的应用,如机器人导航、数字内容生成。
多模态检索系统在具身智能和AI驱动数字内容产业中日益重要,但现有任务复杂度不足,实用性受限。为此我们提出实例驱动的多模态图像检索(IDMR),要求模型在不同语境下检索出与查询图像中同一实例相关的图片,并符合文本描述的场景。不同于关注全局相似性或类别匹配的传统任务,IDMR需在多样场景中保持实例级一致性。为评估该能力,我们基于真实世界物体追踪与第一人称视频数据构建了IDMR-bench。针对训练数据稀缺问题,提出跨域合成方法,从标准检测数据集中裁剪生成55.7万条训练样本。基于多模态大模型(MLLM)的检索模型在120万样本上训练,在传统基准及零样本IDMR-bench上均优于现有方法。实验表明,以往模型在实例感知检索方面存在局限,而MLLM展现出先进检索应用的巨大潜力。完整训练数据集、代码与多种规模模型已开源:https://github.com/BwLiu01/IDMR。
原文摘要 · Abstract (English)
Multimodal retrieval systems are becoming increasingly vital for cutting-edge AI technologies, such as embodied AI and AI-driven digital content industries. However, current multimodal retrieval tasks lack sufficient complexity and demonstrate limited practical application value. It spires us to design Instance-Driven Multimodal Image Retrieval (IDMR), a novel task that requires models to retrieve images containing the same instance as a query image while matching a text-described scenario. Unlike existing retrieval tasks focused on global image similarity or category-level matching, IDMR demands fine-grained instance-level consistency across diverse contexts. To benchmark this capability, we develop IDMR-bench using real-world object tracking and first-person video data. Addressing the scarcity of training data, we propose a cross-domain synthesis method that creates 557K training samples by cropping objects from standard detection datasets. Our Multimodal Large Language Model (MLLM) based retrieval model, trained on 1.2M samples, outperforms state-of-the-art approaches on both traditional benchmarks and our zero-shot IDMR-bench. Experimental results demonstrate previous models' limitations in instance-aware retrieval and highlight the potential of MLLM for advanced retrieval applications. The whole training dataset, codes and models, with wide ranges of sizes, are available at https://github.com/BwLiu01/IDMR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。