将2D检测结果精准转为3D边界框,支持开放世界物体识别。
Boxer: Robust Lifting of Open-World 2D Bounding Boxes to 3D
- 基于Transformer的BoxerNet将2D框升维为3D,融合多视角信息
- 在无深度数据时达0.532 mAP,优于现有方法近50倍
- 适用于开放世界场景,减少对3D标注数据的依赖
三维空间中物体的检测与定位是基础计算机视觉问题。尽管2D目标检测已取得显著进展,但3D定位尤其是开放世界类别仍研究不足且未解决。为此,我们提出Boxer,一种从2D开放词汇检测、图像及可选深度(稀疏点云或稠密深度)中估计静态3D边界框(3DBBs)的算法。核心为BoxerNet,一个基于Transformer的网络,将2D边界框提议提升至3D,再通过多视角融合与几何过滤生成度量空间中全局一致且去重的3DBBs。Boxer利用现有2D检测器(如DETI C、OWLv2、SAM3)进行2D定位,使主模型专注3D升维,降低对昂贵3D标注数据的需求。扩展CuTR框架,引入随机不确定性以增强回归鲁棒性,采用中位数深度块编码支持稀疏深度输入,并在超过120万唯一3DBBs上进行大规模训练。Boxer在开放世界3DBB提升任务中超越当前最优基线:在无稠密深度的自视点设置下,mAP达0.532,远超CuTR的0.010;在提供稠密深度的CA-1M数据集上,mAP为0.412,优于CuTR的0.250。
原文摘要 · Abstract (English)
Detecting and localizing objects in space is a fundamental computer vision problem. While much progress has been made to solve 2D object detection, 3D object localization is much less explored and far from solved, especially for open-world categories. To address this research challenge, we propose Boxer, an algorithm to estimate static 3D bounding boxes (3DBBs) from 2D open-vocabulary object detections, posed images and optional depth either represented as a sparse point cloud or dense depth. At its core is BoxerNet, a transformer-based network which lifts 2D bounding box (2DBB) proposals into 3D, followed by multi-view fusion and geometric filtering to produce globally consistent de-duplicated 3DBBs in metric world space. Boxer leverages the power of existing 2DBB detection algorithms (e.g. DETIC, OWLv2, SAM3) to localize objects in 2D. This allows the main BoxerNet model to focus on lifting to 3D rather than detecting, ultimately reducing the demand for costly annotated 3DBB training data. Extending the CuTR formulation, we incorporate an aleatoric uncertainty for robust regression, a median depth patch encoding to support sparse depth inputs, and large-scale training with over 1.2 million unique 3DBBs. BoxerNet outperforms state-of-the-art baselines in open-world 3DBB lifting, including CuTR in egocentric settings without dense depth (0.532 vs. 0.010 mAP) and on CA-1M with dense depth available (0.412 vs. 0.250 mAP).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。