arXiv:2412.06250cs.CVcs.GR2024-12CVPR被引 31

解决全景图广基线视角合成难题,实现高效真实渲染。

Splatter-360: Generalizable 360$^{\circ}$ Gaussian Splatting for Wide-baseline Panoramic Images

  • 在球面域直接进行多视角匹配,提升几何理解能力。
  • 实测在HM3D和Replica数据集上超越现有方法,支持实时渲染。
  • 适合虚拟现实、仿真等需要高精度全景重建的场景。

广基线全景图像常用于虚拟现实和仿真中,以降低采集成本与存储需求。然而,从这些高分辨率且存在畸变的全景图像中实时合成新视角仍是重大挑战。现有3D高斯点云(3DGS)方法在窄基线下可生成逼真图像,但在广基线全景图像上易过拟合训练视图,因难以从稀疏360°视角中学习精确几何结构。本文提出新型端到端可泛化的3DGS框架Splatter-360,首次在球面域通过球面扫掠算法构建球面代价体积,实现多视角匹配,增强网络深度感知与几何估计能力。同时引入3D感知双投影编码器以缓解全景畸变,并融合跨视角注意力机制提升多视角特征交互。该方法实现鲁棒的3D感知特征表示与实时渲染。在HM3D和Replica数据集上的实验表明,Splatter-360显著优于当前最优的NeRF与3DGS方法(如PanoGRF、MVSplat、DepthSplat和HiSplat),在合成质量与泛化性能方面均有明显提升。代码与训练模型已公开于https://3d-aigc.github.io/Splatter-360/。

原文摘要 · Abstract (English)

Wide-baseline panoramic images are frequently used in applications like VR and simulations to minimize capturing labor costs and storage needs. However, synthesizing novel views from these panoramic images in real time remains a significant challenge, especially due to panoramic imagery's high resolution and inherent distortions. Although existing 3D Gaussian splatting (3DGS) methods can produce photo-realistic views under narrow baselines, they often overfit the training views when dealing with wide-baseline panoramic images due to the difficulty in learning precise geometry from sparse 360$^{\circ}$ views. This paper presents \textit{Splatter-360}, a novel end-to-end generalizable 3DGS framework designed to handle wide-baseline panoramic images. Unlike previous approaches, \textit{Splatter-360} performs multi-view matching directly in the spherical domain by constructing a spherical cost volume through a spherical sweep algorithm, enhancing the network's depth perception and geometry estimation. Additionally, we introduce a 3D-aware bi-projection encoder to mitigate the distortions inherent in panoramic images and integrate cross-view attention to improve feature interactions across multiple viewpoints. This enables robust 3D-aware feature representations and real-time rendering capabilities. Experimental results on the HM3D~\cite{hm3d} and Replica~\cite{replica} demonstrate that \textit{Splatter-360} significantly outperforms state-of-the-art NeRF and 3DGS methods (e.g., PanoGRF, MVSplat, DepthSplat, and HiSplat) in both synthesis quality and generalization performance for wide-baseline panoramic images. Code and trained models are available at \url{https://3d-aigc.github.io/Splatter-360/}.

全景重建3D高斯实时渲染

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。