4K全景图生成新方法,高效且质量高。
PanSplat: 4K Panorama Synthesis with Feed-Forward Gaussian Splatting
- 用球形高斯金字塔和斐波那契排列提升图像质量
- 支持2048×4096分辨率,单张A100 GPU训练
- 适合虚拟现实、自动驾驶等高分辨率应用
随着便携式360°相机的普及,全景图在虚拟现实(VR)、虚拟导览、机器人及自动驾驶等领域受到广泛关注。宽基线全景视图合成成为关键任务,高分辨率、快速推理和内存效率至关重要。然而,现有方法通常受限于较低分辨率(512×1024),因内存与计算需求过高。本文提出PanSplat,一种通用、前馈式方法,可高效支持最高4K(2048×4096)分辨率。其采用定制球形3D高斯金字塔与斐波那契格网布局,在提升图像质量的同时减少信息冗余。为应对高分辨率挑战,提出整合分层球形代价体与局部操作的高斯头结构,实现两步延迟反向传播,使单张A100 GPU上内存高效训练成为可能。实验表明,PanSplat在合成与真实数据集上均达到当前最优效果,兼具卓越效率与图像质量。代码已开源:https://github.com/chengzhag/PanSplat。
原文摘要 · Abstract (English)
With the advent of portable 360° cameras, panorama has gained significant attention in applications like virtual reality (VR), virtual tours, robotics, and autonomous driving. As a result, wide-baseline panorama view synthesis has emerged as a vital task, where high resolution, fast inference, and memory efficiency are essential. Nevertheless, existing methods are typically constrained to lower resolutions (512 $\times$ 1024) due to demanding memory and computational requirements. In this paper, we present PanSplat, a generalizable, feed-forward approach that efficiently supports resolution up to 4K (2048 $\times$ 4096). Our approach features a tailored spherical 3D Gaussian pyramid with a Fibonacci lattice arrangement, enhancing image quality while reducing information redundancy. To accommodate the demands of high resolution, we propose a pipeline that integrates a hierarchical spherical cost volume and Gaussian heads with local operations, enabling two-step deferred backpropagation for memory-efficient training on a single A100 GPU. Experiments demonstrate that PanSplat achieves state-of-the-art results with superior efficiency and image quality across both synthetic and real-world datasets. Code is available at https://github.com/chengzhag/PanSplat.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。