用参考图提升相似物体的视觉定位能力，让AI更准分辨可乐罐。
Multimodal Reference Visual Grounding
- 引入参考图像辅助定位相似物体，结合少样本检测与语言模型匹配。
- 在新数据集上超越Qwen2.5-VL-72B等先进模型，显著提升定位准确率。
- 适合机器人视觉、工业质检等需精准识别相似物品的场景。
视觉定位旨在根据语言描述从图像中检测目标物体。近期大视觉语言模型（LVLMs）通过大规模数据训练显著提升了定位性能，但在输入图像中存在相似物体时仍具挑战性，例如难以区分可口可乐与健怡可乐。若提供两者参考图像，则有助于定位。本文提出新任务——多模态参考视觉定位（MRVG），即模型在拥有数据库中物体参考图像的前提下，结合语言表达从查询图像中定位目标物体。我们构建了首个用于研究MRVG的公开数据集，并提出MRVG-Net方法。该方法通过高效利用参考图像进行少样本检测，结合大语言模型（LLMs）进行物体匹配，相比现有最先进模型如Qwen2.5-VL-72B，在视觉定位任务上表现更优。本方法弥合了少样本检测与视觉定位之间的差距，为机器人等领域的视觉理解拓展新能力。项目页面含视频、代码与数据集：https://irvlutd.github.io/MultiGrounding
原文摘要 · Abstract (English)
Visual grounding focuses on detecting objects from images based on language expressions. Recent Large Vision-Language Models (LVLMs) have significantly advanced visual grounding performance by training large models with large-scale datasets. However, the problem remains challenging, especially when similar objects appear in the input image. For example, an LVLM may not be able to differentiate Diet Coke and regular Coke in an image. In this case, if additional reference images of Diet Coke and regular Coke are available, it can help the visual grounding of similar objects. In this work, we introduce a new task named Multimodal Reference Visual Grounding (MRVG). In this task, a model has access to a set of reference images of objects in a database. Based on these reference images and a language expression, the model is required to detect a target object from a query image. We first introduce a new dataset to study the MRVG problem. Then we introduce a novel method, named MRVG-Net, to solve this visual grounding problem. We show that by efficiently using reference images with few-shot object detection and using Large Language Models (LLMs) for object matching, our method achieves superior visual grounding performance compared to the state-of-the-art LVLMs such as Qwen2.5-VL-72B. Our approach bridges the gap between few-shot detection and visual grounding, unlocking new capabilities for visual understanding, which has wide applications in robotics. Project page with our video, code, and dataset: https://irvlutd.github.io/MultiGrounding
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。