arXiv:2512.09215cs.CV2025-12AAAI被引 5

用图结构分解3D场景,让语言模型逐点推理,提升零样本3D定位准确率。

View-on-Graph: Zero-shot 3D Visual Grounding via Vision-Language Reasoning on Scene Graphs

  • 将3D场景建模为多模态分层图,让语言模型按需访问信息
  • 在ScanNet、MatterPort3D上实现领先零样本性能,优于现有方法
  • 可生成可解释的推理轨迹,适合需要透明性的应用场景

3D视觉定位(3DVG)从语言描述中识别3D场景中的物体。现有零样本方法依赖2D视觉-语言模型(VLM),通过渲染视图或带标记的视频序列将3D空间信息(SI)转换为适配VLM处理的复合输入。但这种VLM + SI范式产生纠缠的视觉表征,迫使VLM处理密集杂乱线索,难以有效利用空间语义关系。本文提出新的VLM × SI范式,将3D SI外化为可增量检索的形式。我们提出视图-图(View-on-Graph, VoG)方法,将场景组织为多模态、多层场景图,使VLM作为主动代理,在遍历场景时选择性获取所需线索。该设计带来双重优势:(i) 将3D上下文结构化为空间与语义一致的场景图,降低VLM推理难度;(ii) 通过主动探索与推理场景图,自然生成可解释的逐步推理轨迹。大量实验表明,VoG在零样本3DVG上达到当前最优性能,验证了结构化场景探索对推进零样本3DVG的潜力。

原文摘要 · Abstract (English)

3D visual grounding (3DVG) identifies objects in 3D scenes from language descriptions. Existing zero-shot approaches leverage 2D vision-language models (VLMs) by converting 3D spatial information (SI) into forms amenable to VLM processing, typically as composite inputs such as specified view renderings or video sequences with overlaid object markers. However, this VLM + SI paradigm yields entangled visual representations that compel the VLM to process entire cluttered cues, making it hard to exploit spatial semantic relationships effectively. In this work, we propose a new VLM x SI paradigm that externalizes the 3D SI into a form enabling the VLM to incrementally retrieve only what it needs during reasoning. We instantiate this paradigm with a novel View-on-Graph (VoG) method, which organizes the scene into a multi-modal, multi-layer scene graph and allows the VLM to operate as an active agent that selectively accesses necessary cues as it traverses the scene. This design offers two intrinsic advantages: (i) by structuring 3D context into a spatially and semantically coherent scene graph rather than confounding the VLM with densely entangled visual inputs, it lowers the VLM's reasoning difficulty; and (ii) by actively exploring and reasoning over the scene graph, it naturally produces transparent, step-by-step traces for interpretable 3DVG. Extensive experiments show that VoG achieves state-of-the-art zero-shot performance, establishing structured scene exploration as a promising strategy for advancing zero-shot 3DVG.

3D视觉定位场景图零样本可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。