无需标注数据,一眼识别任意3D物体与视角下的目标。
Detect Anything 3D in the Wild
- 用2D预训练模型知识迁移支持3D检测
- 在未见类别和新视角下表现领先
- 适合自动驾驶等开放世界应用
尽管深度学习在闭集3D目标检测中取得成功,现有方法在零样本泛化至新物体和相机配置时仍表现不佳。我们提出DetAny3D,一个可提示的3D检测基础模型,仅需单目输入即可在任意相机配置下检测任意新物体。由于3D标注数据稀缺,训练3D基础模型面临根本挑战,因此DetAny3D利用广泛预训练的2D基础模型中蕴含的丰富先验知识来弥补数据不足。为有效实现2D到3D的知识迁移,DetAny3D引入两个核心模块:2D聚合器,用于对齐不同2D基础模型的特征;以及带零嵌入映射的3D解释器,稳定早期2D-to-3D知识迁移过程。实验验证了DetAny3D的强大泛化能力,不仅在未见类别和新相机配置上达到顶尖性能,且在域内数据上也超越多数竞争者。该工作揭示了3D基础模型在真实场景中的潜力,例如自动驾驶中稀有物体检测,并展示了其在开放世界设置下开展更多3D核心任务的前景。更多可视化结果见代码仓库。
原文摘要 · Abstract (English)
Despite the success of deep learning in close-set 3D object detection, existing approaches struggle with zero-shot generalization to novel objects and camera configurations. We introduce DetAny3D, a promptable 3D detection foundation model capable of detecting any novel object under arbitrary camera configurations using only monocular inputs. Training a foundation model for 3D detection is fundamentally constrained by the limited availability of annotated 3D data, which motivates DetAny3D to leverage the rich prior knowledge embedded in extensively pre-trained 2D foundation models to compensate for this scarcity. To effectively transfer 2D knowledge to 3D, DetAny3D incorporates two core modules: the 2D Aggregator, which aligns features from different 2D foundation models, and the 3D Interpreter with Zero-Embedding Mapping, which stabilizes early training in 2D-to-3D knowledge transfer. Experimental results validate the strong generalization of our DetAny3D, which not only achieves state-of-the-art performance on unseen categories and novel camera configurations, but also surpasses most competitors on in-domain data. DetAny3D sheds light on the potential of the 3D foundation model for diverse applications in real-world scenarios, e.g., rare object detection in autonomous driving, and demonstrates promise for further exploration of 3D-centric tasks in open-world settings. More visualization results can be found at our code repository.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。