arXiv:2604.08626cs.CV2026-04被引 9

让3D目标检测能应对真实世界复杂场景,支持多种提示方式并利用深度信息提升精度。

WildDet3D: Scaling Promptable 3D Detection in the Wild

论文配图:WildDet3D: Scaling Promptable 3D Detection in the Wild
图 1 · 摘自论文原文
  • 统一架构支持文本、点、框三种提示,并可实时融合深度信息增强检测
  • 在超百万图像、1.35万类别的新数据集上实现22.6~48.9的AP3D性能
  • 零样本下在Argoverse 2和ScanNet表现突出,深度辅助使平均提升20.7

从单张图像理解三维空间中的物体是空间智能的核心。单目3D目标检测——从输入的RGB图像中恢复物体的范围、位置与朝向——是达成这一目标的关键步骤。为在开放世界中实用,检测器必须超越封闭类别、支持多样提示方式,并在有几何线索时加以利用。当前进展受限于两大瓶颈:现有方法仅针对单一提示类型,缺乏融合额外几何信息的机制;现有3D数据集覆盖类别有限且环境受控,限制了开放世界泛化能力。本文提出两个解决方案:一是构建统一的几何感知架构WildDet3D,原生支持文本、点、框提示,并可在推理时融入辅助深度信号;二是发布目前最大的开放3D检测数据集WildDet3D-Data,基于已有2D标注生成候选3D框,经人工验证后保留,共涵盖超过100万张图像、1.35万个类别,覆盖多样化真实场景。WildDet3D在多个基准上达到新最佳性能,在新开设的WildDet3D-Bench上,文本与框提示分别达22.6/24.8 AP3D;在Omni3D上,对应值为34.2/36.4 AP3D。零样本评估中,于Argoverse 2和ScanNet分别取得40.3/48.9 ODS。值得注意的是,推理时引入深度信息可带来显著增益(跨设置平均提升+20.7 AP)。

原文摘要 · Abstract (English)

Understanding objects in 3D from a single image is a cornerstone of spatial intelligence. A key step toward this goal is monocular 3D object detection--recovering the extent, location, and orientation of objects from an input RGB image. To be practical in the open world, such a detector must generalize beyond closed-set categories, support diverse prompt modalities, and leverage geometric cues when available. Progress is hampered by two bottlenecks: existing methods are designed for a single prompt type and lack a mechanism to incorporate additional geometric cues, and current 3D datasets cover only narrow categories in controlled environments, limiting open-world transfer. In this work we address both gaps. First, we introduce WildDet3D, a unified geometry-aware architecture that natively accepts text, point, and box prompts and can incorporate auxiliary depth signals at inference time. Second, we present WildDet3D-Data, the largest open 3D detection dataset to date, constructed by generating candidate 3D boxes from existing 2D annotations and retaining only human-verified ones, yielding over 1M images across 13.5K categories in diverse real-world scenes. WildDet3D establishes a new state-of-the-art across multiple benchmarks and settings. In the open-world setting, it achieves 22.6/24.8 AP3D on our newly introduced WildDet3D-Bench with text and box prompts. On Omni3D, it reaches 34.2/36.4 AP3D with text and box prompts, respectively. In zero-shot evaluation, it achieves 40.3/48.9 ODS on Argoverse 2 and ScanNet. Notably, incorporating depth cues at inference time yields substantial additional gains (+20.7 AP on average across settings).

3D检测开放世界多模态提示深度融合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。