无需人工标注,实现开放词汇室内3D物体检测。
OpenM3D: Open Vocabulary Multi-view Indoor 3D Object Detection without Human Annotations

- 基于2D图像生成3D伪框,结合图嵌入技术构建连贯结构。
- 在ScanNet200和ARKitScenes上精度超现有方法,单场景推理仅需0.3秒。
- 适合无标注数据环境下快速部署的3D检测任务。
开放词汇3D物体检测是新兴方向,但基于图像的方法研究仍远少于基于点云的方法。本文提出OpenM3D,一种无需人工标注的单阶段多视角室内3D物体检测器,通过继承ImGeoNet的2D诱导体素特征。为支持开放词汇,模型联合训练类无关3D定位损失(需高质量3D伪框)与体素-语义对齐损失(需多样化预训练CLIP特征)。在OV-3DET设定下,仅提供带位姿的RGB-D图像,无3D框或类别标注。我们提出3D伪框生成方法,利用图嵌入将2D分割整合为连贯3D结构,其精度与召回率优于其他方法(包括OV-3DET)。进一步从每个连贯3D结构关联的2D分割中采样多样化的CLIP特征,与体素特征对齐。训练关键在于双损失均指向高质量目标。推理时,OpenM3D仅需多视角图像输入,在ScanNet200与ARKitScenes基准上表现更优,单场景耗时仅0.3秒,超越强基线两阶段方法(使用ViT CLIP分类器与多视图深度估计器),兼具高精度与高速度。
原文摘要 · Abstract (English)
Open-vocabulary (OV) 3D object detection is an emerging field, yet its exploration through image-based methods remains limited compared to 3D point cloud-based methods. We introduce OpenM3D, a novel open-vocabulary multi-view indoor 3D object detector trained without human annotations. In particular, OpenM3D is a single-stage detector adapting the 2D-induced voxel features from the ImGeoNet model. To support OV, it is jointly trained with a class-agnostic 3D localization loss requiring high-quality 3D pseudo boxes and a voxel-semantic alignment loss requiring diverse pre-trained CLIP features. We follow the training setting of OV-3DET where posed RGB-D images are given but no human annotations of 3D boxes or classes are available. We propose a 3D Pseudo Box Generation method using a graph embedding technique that combines 2D segments into coherent 3D structures. Our pseudo-boxes achieve higher precision and recall than other methods, including the method proposed in OV-3DET. We further sample diverse CLIP features from 2D segments associated with each coherent 3D structure to align with the corresponding voxel feature. The key to training a highly accurate single-stage detector requires both losses to be learned toward high-quality targets. At inference, OpenM3D, a highly efficient detector, requires only multi-view images for input and demonstrates superior accuracy and speed (0.3 sec. per scene) on ScanNet200 and ARKitScenes indoor benchmarks compared to existing methods. We outperform a strong two-stage method that leverages our class-agnostic detector with a ViT CLIP-based OV classifier and a baseline incorporating multi-view depth estimator on both accuracy and speed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。