无需训练即可检测未知3D物体,直接用图像生成精准识别结果。
Zoo3D: Zero-Shot 3D Object Detection at Scene Level
- 通过2D实例掩码图聚类生成3D框,结合多视角共识实现语义标注。
- 零样本模式在ScanNet200上达最新性能,超越所有自监督方法。
- 支持带位姿和无位姿图像,可直接部署于真实场景理解任务。
3D目标检测是空间理解的基础。现实环境要求模型能识别多样且未曾见过的物体,而闭集方法在此方面存在明显局限。现有开放词汇3D检测器虽减少标注依赖,但仍需训练场景数据(点云或图像)。本文提出Zoo3D,首个无需训练的3D目标检测框架。方法基于2D实例掩码图聚类构建3D边界框,并利用新型开放词汇模块结合最优视角选择与视角一致性掩码生成进行语义标注。Zoo3D有两种模式:零样本模式Zoo3D$_0$完全无需训练;自监督模式Zoo3D$_1$则在Zoo3D$_0$生成伪标签基础上训练一个类别无关检测器以优化3D框预测。此外,该框架扩展至直接处理有位姿甚至无位姿图像。在ScanNet200与ARKitScenes基准测试中,Zoo3D$_0$与Zoo3D$_1$均达到开放词汇3D检测新纪录。值得注意的是,零样本模式表现优于所有现有自监督方法,证明了无需训练、开箱即用方案在真实3D理解中的强大潜力。代码已开源:https://github.com/col14m/zoo3d。
原文摘要 · Abstract (English)
3D object detection is fundamental for spatial understanding. Real-world environments demand models capable of recognizing diverse, previously unseen objects, which remains a major limitation of closed-set methods. Existing open-vocabulary 3D detectors relax annotation requirements but still depend on training scenes, either as point clouds or images. We take this a step further by introducing Zoo3D, the first training-free 3D object detection framework. Our method constructs 3D bounding boxes via graph clustering of 2D instance masks, then assigns semantic labels using a novel open-vocabulary module with best-view selection and view-consensus mask generation. Zoo3D operates in two modes: the zero-shot Zoo3D$_0$, which requires no training at all, and the self-supervised Zoo3D$_1$, which refines 3D box prediction by training a class-agnostic detector on Zoo3D$_0$-generated pseudo labels. Furthermore, we extend Zoo3D beyond point clouds to work directly with posed and even unposed images. Across ScanNet200 and ARKitScenes benchmarks, both Zoo3D$_0$ and Zoo3D$_1$ achieve state-of-the-art results in open-vocabulary 3D object detection. Remarkably, our zero-shot Zoo3D$_0$ outperforms all existing self-supervised methods, hence demonstrating the power and adaptability of training-free, off-the-shelf approaches for real-world 3D understanding. Code is available at https://github.com/col14m/zoo3d .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。