仅用普通照片实现室内布局与物体3D定位，无需相机姿态或深度数据。
TUN3D: Towards Real-World Scene Understanding from Unposed Images
- 用多视角图像和新墙面表示法联合推断布局与物体3D位置
- 在三个基准上均达顶尖性能，尤其布局估计显著领先
- 适合无深度相机的现实场景理解应用
室内场景理解中的布局估计与3D物体检测是两大基础任务。二者结合可生成紧凑且语义丰富的空间表征。现有方法通常依赖点云输入，但多数消费级相机无深度传感器，纯视觉数据更常见。我们提出TUN3D，首个仅以多视角图像为输入，不需真实相机姿态或深度监督，即可实现联合布局估计与3D物体检测的方法。该方法采用轻量级稀疏卷积主干网络，配备两个专用分支：一个用于3D物体检测，一个用于布局估计，并引入新颖有效的参数化墙面表示。大量实验表明，TUN3D在三个挑战性场景理解基准上均达到当前最优性能：(i) 使用真实点云，(ii) 使用已对齐图像，(iii) 使用未对齐图像。其3D物体检测表现与专用方法相当，但布局估计显著提升，建立了整体室内场景理解的新标杆。代码已开源：https://github.com/col14m/tun3d。
原文摘要 · Abstract (English)
Layout estimation and 3D object detection are two fundamental tasks in indoor scene understanding. When combined, they enable the creation of a compact yet semantically rich spatial representation of a scene. Existing approaches typically rely on point cloud input, which poses a major limitation since most consumer cameras lack depth sensors and visual-only data remains far more common. We address this issue with TUN3D, the first method that tackles joint layout estimation and 3D object detection in real scans, given multi-view images as input, and does not require ground-truth camera poses or depth supervision. Our approach builds on a lightweight sparse-convolutional backbone and employs two dedicated heads: one for 3D object detection and one for layout estimation, leveraging a novel and effective parametric wall representation. Extensive experiments show that TUN3D achieves state-of-the-art performance across three challenging scene understanding benchmarks: (i) using ground-truth point clouds, (ii) using posed images, and (iii) using unposed images. While performing on par with specialized 3D object detection methods, TUN3D significantly advances layout estimation, setting a new benchmark in holistic indoor scene understanding. Code is available at https://github.com/col14m/tun3d .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。