arXiv:2411.02229cs.CV2024-11NeurIPS被引 26

少样本下精准生成新视角,用多阶段训练提升渲染质量

FewViewGS: Gaussian Splatting with Few View Matching and Multi-stage Training

  • 通过图像匹配约束新视角生成,无需预训练深度或扩散模型
  • 在真实与合成数据集上少样本表现优于现有方法
  • 保留局部颜色结构,有效减少渲染伪影,适合低资源场景

从图像进行新视角合成领域因神经辐射场(NeRF)和最近的3D高斯点阵(Gaussian Splatting)而迅速发展。尽管高斯点阵在充足训练图像下表现优异,但其无结构显式表示在稀疏输入时易过拟合,导致渲染性能下降。为此,我们提出一种基于3D高斯的新视角合成方法,适用于稀疏输入图像,能准确渲染训练视角之外的新视角。该方法采用多阶段训练策略,通过已有训练图像的匹配关系,在不依赖预训练深度估计或扩散模型的前提下,对训练帧之间的采样新视角施加一致性约束,结合颜色、几何和语义损失进行监督。此外,引入保持局部性正则化,通过保留场景局部颜色结构消除渲染伪影。在合成与真实世界数据集上的评估表明,本方法在少样本新视角合成任务中达到或超过现有最先进水平。

原文摘要 · Abstract (English)

The field of novel view synthesis from images has seen rapid advancements with the introduction of Neural Radiance Fields (NeRF) and more recently with 3D Gaussian Splatting. Gaussian Splatting became widely adopted due to its efficiency and ability to render novel views accurately. While Gaussian Splatting performs well when a sufficient amount of training images are available, its unstructured explicit representation tends to overfit in scenarios with sparse input images, resulting in poor rendering performance. To address this, we present a 3D Gaussian-based novel view synthesis method using sparse input images that can accurately render the scene from the viewpoints not covered by the training images. We propose a multi-stage training scheme with matching-based consistency constraints imposed on the novel views without relying on pre-trained depth estimation or diffusion models. This is achieved by using the matches of the available training images to supervise the generation of the novel views sampled between the training frames with color, geometry, and semantic losses. In addition, we introduce a locality preserving regularization for 3D Gaussians which removes rendering artifacts by preserving the local color structure of the scene. Evaluation on synthetic and real-world datasets demonstrates competitive or superior performance of our method in few-shot novel view synthesis compared to existing state-of-the-art methods.

新视角合成3D高斯少样本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。