用单张全景图实现360度场景三维重建,效果超越现有方法。
Unified Panoramic Geometry Estimation via Multi-View Foundation Models

- 基于预训练3D模型,统一处理透视图与全景图。
- 单次前向传播预测深度、法向量和天空掩码,精度领先。
- 无需微调即可跨场景通用,适合实时全景建模应用。
从透视图像中进行几何估计已取得显著进展,现已有现成的3D基础模型能仅凭多视角图像甚至单张图像重建三维场景结构。自然延伸是利用全景图实现360度场景重建,有望仅凭一张全景图恢复完整空间结构。本文提出PaGeR(Panoramic Geometry Reconstruction)框架,将专为透视图像设计的3D基础模型迁移至全景领域。策略是基于预训练的3D重建Transformer,通过最小化架构修改并混合透视与全景图像训练,使其在单次前向传播中同时输出尺度无关深度、真实尺度深度、表面法向量和天空掩码,适用于透视与全景图像。该方法保留了原始模型丰富的3D先验知识,同时学习了从单张全景图生成几何一致的360度场景的能力。我们在室内外多种环境中广泛测试,结果表明其在多个场景下均达到当前最优性能,并具备优异的零样本泛化能力。代码、数据与模型已公开。
原文摘要 · Abstract (English)
Geometry estimation from perspective images has greatly advanced, maturing to the point where off-the-shelf foundation models are able to reconstruct 3D scene structure not only from multi-view imagery, but even from a single view. A natural extension is 3D reconstruction from panoramas, with the exciting prospect of recovering a full 360-degree scene from a single panoramic image. In this work, we introduce PaGeR (Panoramic Geometry Reconstruction), a framework to lift powerful 3D foundation models designed for perspective imagery to the panorama domain. Our strategy is to start from a pre-trained transformer for 3D reconstruction and turn it into a unified high-performance model that predicts scale-invariant depth, metric depth, surface normals, and sky masks from both perspective and omnidirectional images, in a single forward pass. By keeping architectural changes to a minimum and mixing perspective and panoramic images during training, PaGeR retains the rich 3D prior of the underlying foundation model while learning to also estimate geometrically consistent 360-degree scenes from single panoramas. We extensively test our method in both indoor and outdoor environments and find that it delivers state-of-the-art performance and excellent zero-shot performance across a wide range of scenes. Code, data and models are available $\href{https://github.com/prs-eth/PaGeR}{\text{here}}$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。