用语言模型直接在点云上做零样本3D物体定位，更准更快。
AgentGrounder: Zero-Shot 3D Visual Pointcloud Grounding using Multimodal Language Models
- 通过构建对象表，仅检索相关候选，减少错误传播。
- 在ScanRefer和Nr3D上分别提升2.5%和6.3%准确率。
- 适合需要灵活、低误差3D视觉定位的智能体应用。
3D视觉定位是具身智能的关键能力，要求智能体根据自然语言描述定位3D场景中的物体。现有零样本方法依赖2D视觉语言模型，常需多视角图像，且受限于标准3D分割工具提供的语义与空间细节不足。本文提出AgentGrounder，一种直接在彩色点云上运行的零样本3D视觉定位框架，无需特定任务的3D训练。其采用两阶段设计：(1) 离线阶段使用3D模型构建包含实例ID、语义标签和3D边界框的物体查找表（OLT）；(2) 在线工具驱动的智能体将查询分解，从OLT中检索相关候选，进行几何评分，并在需要额外视觉证据（如颜色、材质或视点敏感线索）时按需触发图像渲染。相比固定锚点-目标匹配流程，该设计降低级联匹配误差，提升上下文窗口效率。在ScanRefer和Nr3D的零样本设置下评估，结果优于SeeGround，ScanRefer上[email protected]提升2.5%，Nr3D上提升6.3%，其中视点无关查询提升达6.3%。表明选择性检索、几何推理与自适应视觉检查结合，可构建实用且鲁棒的开放词汇3D定位基础。
原文摘要 · Abstract (English)
3D Visual Grounding (3DVG) is an essential capability for embodied AI, requiring agents to localize objects in 3D scenes based on natural language descriptions. Recent zero-shot methods leverage 2D vision-language models (LVLMs). However, they often rely on existing sets of multi-view images and struggle with the limited semantic and spatial details provided by standard 3D segmentation tools. We present $\textbf{AgentGrounder}$, a zero-shot 3D visual grounding framework that operates directly on colored point clouds without task-specific 3D training. Our approach follows a two-stage design: (1) an offline stage that applies 3D model to build an Object Lookup Table (OLT) with instance IDs, semantic labels, 3D bounding boxes; and (2) an online tool-driven agent that decomposes each query, retrieves only relevant candidates from the OLT, performs geometric scoring, and triggers image rendering on demand when additional visual evidence (e.g., color, material, or viewpoint-sensitive cues) is required. Compared with fixed anchor-target matching pipelines, this design reduces cascading matching errors and improves context-window efficiency by avoiding prompts overloaded with irrelevant objects. We evaluate on ScanRefer and Nr3D under a zero-shot setting and observe consistent improvements over SeeGround in our setup, including +2.5% [email protected] on ScanRefer and +6.3% on Nr3D, with a notable +6.3% gain on Nr3D view-independent queries. These results show that combining selective retrieval, geometric reasoning, and adaptive visual inspection yields a practical and robust foundation for open-vocabulary 3D grounding. Our code is available at https://github.com/be2rlab/AgentGrounder.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。