提出无需网格的3D实例分割框架,支持复杂查询和跨场景泛化。
OpenTrack3D: Towards Accurate and Generalizable Open-Vocabulary 3D Instance Segmentation
- 在线构建跨视角一致的3D目标提议,不依赖预训练网格或特定数据集模型。
- 在ScanNet200等四个基准上实现领先性能,支持复杂语义查询。
- 适合机器人、AR/VR等需泛化能力的真实场景应用。
将开放词汇3D实例分割(OV-3DIS)推广至多样、非结构化且无网格的环境,对机器人与增强现实/虚拟现实至关重要,但仍是重大挑战。现有方法存在两大局限:一是提议生成依赖数据集特定的提案网络或基于网格的超点,无法适用于无网格场景,限制了新场景的泛化能力;二是基于CLIP的分类器文本推理能力弱,难以识别复合性与功能性用户查询。为此,我们提出OpenTrack3D,一种通用且准确的框架。不同于依赖预生成提案的方法,OpenTrack3D采用新型视觉-空间追踪器在线构建跨视角一致的物体提议。给定RGB-D流,系统首先利用2D开放词汇分割器生成掩码,并通过深度图将其提升至3D点云。利用DINO特征图提取掩码引导的实例特征,追踪器融合视觉与空间线索以保持实例一致性。整个流程完全无网格,但当场景网格可用时也提供可选的超点精修模块以进一步提升性能。最后,我们用多模态大语言模型(MLLM)替代CLIP,显著增强对复杂用户查询的组合推理能力。在ScanNet200、Replica、ScanNet++和SceneFun3D等多个基准上的大量实验表明,该方法达到当前最优性能并具备强大泛化能力。
原文摘要 · Abstract (English)
Generalizing open-vocabulary 3D instance segmentation (OV-3DIS) to diverse, unstructured, and mesh-free environments is crucial for robotics and AR/VR, yet remains a significant challenge. We attribute this to two key limitations of existing methods: (1) proposal generation relies on dataset-specific proposal networks or mesh-based superpoints, rendering them inapplicable in mesh-free scenarios and limiting generalization to novel scenes; and (2) the weak textual reasoning of CLIP-based classifiers, which struggle to recognize compositional and functional user queries. To address these issues, we introduce OpenTrack3D, a generalizable and accurate framework. Unlike methods that rely on pre-generated proposals, OpenTrack3D employs a novel visual-spatial tracker to construct cross-view consistent object proposals online. Given an RGB-D stream, our pipeline first leverages a 2D open-vocabulary segmenter to generate masks, which are lifted to 3D point clouds using depth. Mask-guided instance features are then extracted using DINO feature maps, and our tracker fuses visual and spatial cues to maintain instance consistency. The core pipeline is entirely mesh-free, yet we also provide an optional superpoints refinement module to further enhance performance when scene mesh is available. Finally, we replace CLIP with a multi-modal large language model (MLLM), significantly enhancing compositional reasoning for complex user queries. Extensive experiments on diverse benchmarks, including ScanNet200, Replica, ScanNet++, and SceneFun3D, demonstrate state-of-the-art performance and strong generalization capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。