arXiv:2603.00912cs.CV2026-03中稿 · CVPR被引 5

无需相机位姿,用图像内蕴含的3D先验实现多视角室内物体检测。

VGGT-Det: Mining VGGT Internal Priors for Sensor-Geometry-Free Multi-View Indoor 3D Object Detection

  • 利用VGGT内部注意力图生成初始查询,聚焦物体区域并保持空间结构。
  • 通过可学习的'看-查'机制动态聚合跨层几何特征,实现2D到3D的渐进提升。
  • 在无传感器几何输入下,比现有方法提升4.4和8.6点mAP,适合真实场景部署。

当前多视角室内3D物体检测依赖昂贵的传感器几何信息(如精确标定的多视角相机位姿)来融合多视图信息构建全局场景表示,限制了实际应用。本文提出更实用的无传感器几何(SG-Free)多视角室内3D物体检测设定,即不依赖相机位姿或深度数据。基于视觉几何基础变压器(VGGT)能直接从图像中推断强3D线索的发现,我们提出首个专为该设定设计的框架VGGT-Det。不同于简单使用VGGT输出,本方法将VGGT编码器融入基于Transformer的流水线,并引入两个新组件:(i) 注意力引导查询生成(AG),利用VGGT注意力图作为语义先验初始化物体查询,提升定位精度并保留全局空间结构;(ii) 查询驱动特征聚合(QD),通过可学习的See-Query与物体查询交互,主动‘查看’其所需内容,并动态聚合跨多层的几何特征,逐步将2D特征升维至3D。实验表明,VGGT-Det在ScanNet和ARKitScenes上分别比最佳现有方法提升4.4和8.6 [email protected]。消融实验证明,本方法有效利用了VGGT内部学习到的语义与几何先验。

原文摘要 · Abstract (English)

Current multi-view indoor 3D object detectors rely on sensor geometry that is costly to obtain (i.e., precisely calibrated multi-view camera poses) to fuse multi-view information into a global scene representation, limiting deployment in real-world scenes. We target a more practical setting: Sensor-Geometry-Free (SG-Free) multi-view indoor 3D object detection, where there are no sensor-provided geometric inputs (multi-view poses or depth). Recent Visual Geometry Grounded Transformer (VGGT) shows that strong 3D cues can be inferred directly from images. Building on this insight, we present VGGT-Det, the first framework tailored for SG-Free multi-view indoor 3D object detection. Rather than merely consuming VGGT predictions, our method integrates VGGT encoder into a transformer-based pipeline. To effectively leverage both the semantic and geometric priors from inside VGGT, we introduce two novel key components: (i) Attention-Guided Query Generation (AG): exploits VGGT attention maps as semantic priors to initialize object queries, improving localization by focusing on object regions while preserving global spatial structure; (ii) Query-Driven Feature Aggregation (QD): a learnable See-Query interacts with object queries to 'see' what they need, and then dynamically aggregates multi-level geometric features across VGGT layers that progressively lift 2D features into 3D. Experiments show that VGGT-Det significantly surpasses the best-performing method in the SG-Free setting by 4.4 and 8.6 [email protected] on ScanNet and ARKitScenes, respectively. Ablation study shows that VGGT's internally learned semantic and geometric priors can be effectively leveraged by our AG and QD.

3D检测无几何输入视觉先验Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。