arXiv:2508.20758cs.CVcs.AI2025-08被引 12

用多视角图像增强3D物体定位,实现零样本场景下更精准的自然语言定位。

SeqVLM: Proposal-Guided Multi-View Sequences Reasoning via VLM for Zero-Shot 3D Visual Grounding

  • 通过生成候选3D实例并投影到多视角图像,保留空间关系与上下文细节。
  • 在ScanRefer和Nr3D上分别达到55.6%和53.2%的[email protected],领先现有方法4.0%和5.2%。
  • 适合希望在无训练场景中实现高泛化3D视觉定位的研究者或应用开发者。

3D视觉定位(3DVG)旨在使用自然语言描述定位3D场景中的物体。尽管监督方法在受限环境下表现更好,但零样本3DVG因无需特定场景训练,在真实应用中更具潜力。然而,现有零样本方法受限于单视角定位,存在空间推理范围小、上下文遗漏或细节退化等问题。为此,本文提出SeqVLM,一种基于多视角真实场景图像与空间信息的零样本3DVG框架。首先,通过3D语义分割网络生成3D实例候选,并经语义过滤保留相关候选;再采用提案引导的多视角投影策略,将候选投影至真实场景图像序列,保持3D点云转图像过程中的空间关系与上下文细节。为缓解视觉语言模型(VLM)计算负担,设计动态调度机制,迭代处理序列-查询提示,利用VLM跨模态推理能力定位文本指定物体。在ScanRefer和Nr3D基准测试中,该方法分别取得55.6%和53.2%的[email protected],较此前零样本方法提升4.0%和5.2%,推动3DVG向更高泛化性与真实应用迈进。代码已开源。

原文摘要 · Abstract (English)

3D Visual Grounding (3DVG) aims to localize objects in 3D scenes using natural language descriptions. Although supervised methods achieve higher accuracy in constrained settings, zero-shot 3DVG holds greater promise for real-world applications since eliminating scene-specific training requirements. However, existing zero-shot methods face challenges of spatial-limited reasoning due to reliance on single-view localization, and contextual omissions or detail degradation. To address these issues, we propose SeqVLM, a novel zero-shot 3DVG framework that leverages multi-view real-world scene images with spatial information for target object reasoning. Specifically, SeqVLM first generates 3D instance proposals via a 3D semantic segmentation network and refines them through semantic filtering, retaining only semantic-relevant candidates. A proposal-guided multi-view projection strategy then projects these candidate proposals onto real scene image sequences, preserving spatial relationships and contextual details in the conversion process of 3D point cloud to images. Furthermore, to mitigate VLM computational overload, we implement a dynamic scheduling mechanism that iteratively processes sequances-query prompts, leveraging VLM's cross-modal reasoning capabilities to identify textually specified objects. Experiments on the ScanRefer and Nr3D benchmarks demonstrate state-of-the-art performance, achieving [email protected] scores of 55.6% and 53.2%, surpassing previous zero-shot methods by 4.0% and 5.2%, respectively, which advance 3DVG toward greater generalization and real-world applicability. The code is available at https://github.com/JiawLin/SeqVLM.

3D视觉定位零样本多视角视觉语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。