单目视频实时全景3D重建新框架，速度与精度兼优。
EPRecon: An Efficient Framework for Real-Time Panoptic 3D Reconstruction from Monocular Video
- 直接在3D体素中生成占用概率估计深度先验，提升效率
- 融合体素与图像特征提取语义信息，分割更精准
- 实测在ScanNetV2上实现实时重建，性能领先
从单目视频进行全景3D重建是机器人场景理解中的基础感知任务。然而，现有方法在推理速度和准确性方面存在效率瓶颈，限制了实际应用。本文提出EPRecon，一种高效实时的全景3D重建框架。现有基于体素的方法通常通过多视图深度图融合获取场景深度先验，该过程耗时且难以支持实时重建。为此，我们设计了一个轻量级模块，直接在3D体素中估计深度先验，通过生成所有体素的占用概率来提升重建质量。此外，相比现有全景分割方法，EPRecon同时从体素特征和对应图像特征中提取全景特征，获得更细致、全面的实例级语义信息，从而实现更高精度的分割结果。在ScanNetV2数据集上的实验表明，EPRecon在全景3D重建质量和实时推理能力方面均优于当前最先进方法。代码已开源：https://github.com/zhen6618/EPRecon。
原文摘要 · Abstract (English)
Panoptic 3D reconstruction from a monocular video is a fundamental perceptual task in robotic scene understanding. However, existing efforts suffer from inefficiency in terms of inference speed and accuracy, limiting their practical applicability. We present EPRecon, an efficient real-time panoptic 3D reconstruction framework. Current volumetric-based reconstruction methods usually utilize multi-view depth map fusion to obtain scene depth priors, which is time-consuming and poses challenges to real-time scene reconstruction. To address this issue, we propose a lightweight module to directly estimate scene depth priors in a 3D volume for reconstruction quality improvement by generating occupancy probabilities of all voxels. In addition, compared with existing panoptic segmentation methods, EPRecon extracts panoptic features from both voxel features and corresponding image features, obtaining more detailed and comprehensive instance-level semantic information and achieving more accurate segmentation results. Experimental results on the ScanNetV2 dataset demonstrate the superiority of EPRecon over current state-of-the-art methods in terms of both panoptic 3D reconstruction quality and real-time inference. Code is available at https://github.com/zhen6618/EPRecon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。