Stream3Dv2实现高效零样本3D场景理解,抗噪且实时响应。
Stream3Dv2: Geometric-Semantic Fusion Enhanced Streaming Zero-Shot 3D Scene Understanding

- 采用局部到历史的嵌套结构,低延迟处理连续RGB-D流数据。
- 融合几何与语义信息,显著提升分割准确率,优于现有基线。
- 适合需要实时3D感知的机器人、AR/VR等开放世界应用。
近期,基于视觉基础模型的开放词汇零样本3D场景理解成为数据密集型监督方法的有前景替代方案。然而,其在真实场景中的部署严重受限于无法高效处理流式RGB-D输入,以及对2D分割掩码噪声的固有敏感性。为此,我们提出Stream3Dv2,一种无需训练的新型框架,用于鲁棒的流式3D感知。Stream3Dv2通过原创的嵌套局部-历史架构处理序列数据,捕捉多视角一致性,同时避免高计算开销,支持及时响应。核心上,我们引入全面的几何-语义融合机制,通过显式利用语义引导,将3D分割建模为点集合并与划分问题,以缓解几何噪声与语义模糊。此外,提出基于流形距离的点云优化策略:利用局部流形图进行点到流形优化,解决欧氏距离度量导致的边界分割失败;并采用几何包围盒动态激活与更新历史实例,实现快速流形间精化。在公开数据集上的大量实验表明,Stream3Dv2在基础开放词汇流式3D分割与检测任务中持续优于现有基线。最后,我们将该框架与基于LLM的智能体集成,实现了先进的语言驱动3D场景理解,凸显其在开放世界具身智能中的潜力。代码将在https://github.com/SubmissionsIn/Stream3D更新。
原文摘要 · Abstract (English)
Recently, open-vocabulary zero-shot 3D scene understanding using vision foundation models has emerged as a promising alternative to data-intensive supervised methods. However, deploying these models in real-world scenarios is severely hindered by their inability to efficiently handle streaming RGB-D inputs and their inherent vulnerability to noise 2D segmentation masks. To address these critical limitations, we propose Stream3Dv2, a novel training-free framework designed for robust streaming 3D perception. Stream3Dv2 processes sequential data through an original nested local-to-historical architecture, capturing multi-view consistency while circumventing the high computational overhead so as to support timely responses. At its core, we introduce a comprehensive geometric-semantic fusion mechanism that resolves geometric noise and semantic ambiguity by explicitly utilizing semantic guidance and formulating 3D segmentation as solving point-and-set merging and partitioning problems. Furthermore, we present an innovative manifold-distance-based point cloud refinement strategy. This approach leverages local manifold graphs for point-to-manifold optimization that mitigates the boundary delineation failures caused by Euclidean-distance metrics, and employs geometric bounding boxes to dynamically activate and update historical instances for achieving rapid manifold-to-manifold refinement. Extensive experiments on public datasets demonstrate that Stream3Dv2 consistently outperforms existing baselines in foundational open-vocabulary streaming 3D segmentation and detection. Finally, we show that integrating our framework with an LLM-based agent enables advanced language-driven 3D scene understanding, underscoring its potential for open-world embodied intelligence. Code will be updated at https://github.com/SubmissionsIn/Stream3D.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。