用2D模型+3D重建实现开放词汇的3D实例分割,提升小类性能。
OVSeg3R: Learn Open-vocabulary Instance Segmentation from 2D via 3D Reconstruction
- 从2D视频重建3D场景,投影2D掩码生成3D标注
- 新算法解决部分标注导致的误检,使训练更稳定
- 结合2D边界信息聚类超点,更好分割非显著物体
本文提出OVSeg3R训练方案,通过2D感知模型与3D重建结合,实现开放词汇的3D实例分割。该方法直接使用2D视频生成的3D场景作为输入,避免人工调整,贴近真实应用。利用2D到3D的对应关系,将开放词汇2D模型的实例掩码投影至3D,生成对应子场景的标注。为防止因2D到3D部分标注引入错误正例,提出视图级实例分割算法,按视图划分预测以稳定训练。此外,针对3D重建常过度平滑几何细节的问题,提出2D实例边界感知超点,利用2D掩码约束聚类,防止超点跨实例边界。该设计使主流封闭词汇3D实例分割模型扩展至开放词汇,并显著缩小头尾类别差距,在ScanNet200上整体提升+2.3 mAP;在标准开放词汇设置下,对新类别性能提升约+7.1 mAP,验证了有效性。
原文摘要 · Abstract (English)
In this paper, we propose a training scheme called OVSeg3R to learn open-vocabulary 3D instance segmentation from well-studied 2D perception models with the aid of 3D reconstruction. OVSeg3R directly adopts reconstructed scenes from 2D videos as input, avoiding costly manual adjustment while aligning input with real-world applications. By exploiting the 2D to 3D correspondences provided by 3D reconstruction models, OVSeg3R projects each view's 2D instance mask predictions, obtained from an open-vocabulary 2D model, onto 3D to generate annotations for the view's corresponding sub-scene. To avoid incorrectly introduced false positives as supervision due to partial annotations from 2D to 3D, we propose a View-wise Instance Partition algorithm, which partitions predictions to their respective views for supervision, stabilizing the training process. Furthermore, since 3D reconstruction models tend to over-smooth geometric details, clustering reconstructed points into representative super-points based solely on geometry, as commonly done in mainstream 3D segmentation methods, may overlook geometrically non-salient objects. We therefore introduce 2D Instance Boundary-aware Superpoint, which leverages 2D masks to constrain the superpoint clustering, preventing superpoints from violating instance boundaries. With these designs, OVSeg3R not only extends a state-of-the-art closed-vocabulary 3D instance segmentation model to open-vocabulary, but also substantially narrows the performance gap between tail and head classes, ultimately leading to an overall improvement of +2.3 mAP on the ScanNet200 benchmark. Furthermore, under the standard open-vocabulary setting, OVSeg3R surpasses previous methods by about +7.1 mAP on the novel classes, further validating its effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。