用100万标注数据训练3D检测模型,让手机拍的室内场景也能精准识物。
Cubify Anything: Scaling Indoor 3D Object Detection
- 直接从2D图像特征预测3D框,跳过点云/体素处理
- 在100万级数据上召回率超62%,抗噪能力更强
- 适合做移动设备上的实时3D感知,尤其适合无深度传感器场景
本文针对手持设备获取的单张RGB(-D)图像进行室内3D物体检测,旨在推动数据与建模两方面的突破。现有数据集在规模、精度和物体多样性上存在明显局限,为此提出Cubify-Anything 1M(CA-1M)数据集,在超过1000个高精度激光扫描场景中对40多万个3D物体进行详尽标注,并与超过3500个手持式、第一视角采集的数据实现近乎完美的配准。进一步提出完全基于Transformer的检测基线模型CuTR,不依赖点云或体素的3D表示,而是直接从RGB(-D)输入提取的2D特征预测3D边界框。尽管缺乏3D先验知识,但结合CA-1M训练后,CuTR在3D物体召回率上超过62%,显著优于传统点云方法,且对商品级激光雷达生成的深度图噪声具有更强鲁棒性;同时在仅使用RGB的情况下无需修改架构即可取得良好表现。此外,基于CA-1M预训练的CuTR在更丰富的SUN RGB-D变体上也超越点云方法,表明:当数据量充足时,传统3D归纳偏置反而成为瓶颈。整体证明我们正迈向能‘万物立方化’的智能感知时代。
原文摘要 · Abstract (English)
We consider indoor 3D object detection with respect to a single RGB(-D) frame acquired from a commodity handheld device. We seek to significantly advance the status quo with respect to both data and modeling. First, we establish that existing datasets have significant limitations to scale, accuracy, and diversity of objects. As a result, we introduce the Cubify-Anything 1M (CA-1M) dataset, which exhaustively labels over 400K 3D objects on over 1K highly accurate laser-scanned scenes with near-perfect registration to over 3.5K handheld, egocentric captures. Next, we establish Cubify Transformer (CuTR), a fully Transformer 3D object detection baseline which rather than operating in 3D on point or voxel-based representations, predicts 3D boxes directly from 2D features derived from RGB(-D) inputs. While this approach lacks any 3D inductive biases, we show that paired with CA-1M, CuTR outperforms point-based methods - accurately recalling over 62% of objects in 3D, and is significantly more capable at handling noise and uncertainty present in commodity LiDAR-derived depth maps while also providing promising RGB only performance without architecture changes. Furthermore, by pre-training on CA-1M, CuTR can outperform point-based methods on a more diverse variant of SUN RGB-D - supporting the notion that while inductive biases in 3D are useful at the smaller sizes of existing datasets, they fail to scale to the data-rich regime of CA-1M. Overall, this dataset and baseline model provide strong evidence that we are moving towards models which can effectively Cubify Anything.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。