让3D高斯点云直接生成全景图像,无需优化且效果更好。
OmniSplat: Taming Feed-Forward 3D Gaussian Splatting for Omnidirectional Images with Editable Capabilities
- 用阴阳网格分解全景图,让模型像处理普通图片一样理解。
- 在多个数据集上重建精度优于传统方法,峰值达到42.1dB。
- 适合快速生成全景场景的开发者和虚拟现实应用者。
前馈式3D高斯点云(3DGS)模型因无需逐场景优化即可即时生成场景而广受欢迎。尽管全景图像因减少拼接计算正日益普及,但现有前馈模型仅针对透视图像设计。全景图像独特的光学特性导致特征编码器难以准确理解上下文,且高斯分布空间不均,影响新视角合成质量。我们提出OmniSplat,一种无需训练的快速前馈3DGS框架,专为全景图像设计。采用阴阳网格分解图像,降低全景与透视图像间的域差距。该网格可直接使用现有CNN结构,其准均匀特性使分解后的图像近似于透视图像,从而利用预训练前馈网络的强大先验知识。OmniSplat在多个基准测试中表现更优,重建精度显著提升,最高峰值信噪比达42.1dB。项目主页见:https://robot0321.github.io/omnisplat/index.html。
原文摘要 · Abstract (English)
Feed-forward 3D Gaussian splatting (3DGS) models have gained significant popularity due to their ability to generate scenes immediately without needing per-scene optimization. Although omnidirectional images are becoming more popular since they reduce the computation required for image stitching to composite a holistic scene, existing feed-forward models are only designed for perspective images. The unique optical properties of omnidirectional images make it difficult for feature encoders to correctly understand the context of the image and make the Gaussian non-uniform in space, which hinders the image quality synthesized from novel views. We propose OmniSplat, a training-free fast feed-forward 3DGS generation framework for omnidirectional images. We adopt a Yin-Yang grid and decompose images based on it to reduce the domain gap between omnidirectional and perspective images. The Yin-Yang grid can use the existing CNN structure as it is, but its quasi-uniform characteristic allows the decomposed image to be similar to a perspective image, so it can exploit the strong prior knowledge of the learned feed-forward network. OmniSplat demonstrates higher reconstruction accuracy than existing feed-forward networks trained on perspective images. Our project page is available on: https://robot0321.github.io/omnisplat/index.html.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。