用2D图像实现零样本3D目标定位，不依赖3D数据或先验知识。
VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Grounding
- 基于视觉语言模型，仅用2D图像动态拼接和多视角投影定位目标。
- 在ScanRefer上达51.6% [email protected]，Nr3D上达48.0%准确率，超越现有零样本方法。
- 适合无3D标注数据的机器人场景理解，尤其适用于复杂语言查询。
3D视觉定位对机器人至关重要，需融合自然语言与3D场景理解。传统依赖3D点云监督学习的方法受限于数据稀缺。近期零样本方法利用大语言模型缓解数据问题，但仅依赖对象中心信息，难以处理复杂查询。本文提出VLM-Grounder，一种仅基于2D图像的零样本3D视觉定位新框架。该方法动态拼接图像序列，采用定位-反馈机制识别目标，并通过多视角集成投影精确估计3D边界框。在ScanRefer和Nr3D数据集上的实验表明，VLM-Grounder优于先前零样本方法，在ScanRefer上达到51.6% [email protected]，Nr3D上达到48.0%准确率，且无需3D几何结构或对象先验。代码已开源。
原文摘要 · Abstract (English)
3D visual grounding is crucial for robots, requiring integration of natural language and 3D scene understanding. Traditional methods depending on supervised learning with 3D point clouds are limited by scarce datasets. Recently zero-shot methods leveraging LLMs have been proposed to address the data issue. While effective, these methods only use object-centric information, limiting their ability to handle complex queries. In this work, we present VLM-Grounder, a novel framework using vision-language models (VLMs) for zero-shot 3D visual grounding based solely on 2D images. VLM-Grounder dynamically stitches image sequences, employs a grounding and feedback scheme to find the target object, and uses a multi-view ensemble projection to accurately estimate 3D bounding boxes. Experiments on ScanRefer and Nr3D datasets show VLM-Grounder outperforms previous zero-shot methods, achieving 51.6% [email protected] on ScanRefer and 48.0% Acc on Nr3D, without relying on 3D geometry or object priors. Codes are available at https://github.com/OpenRobotLab/VLM-Grounder .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。