arXiv:2410.08941cs.CV2024-10被引 34

将高斯点云与网格结合,实现更真实、更高效的三维场景渲染。

MeshGS: Adaptive Mesh-Aligned Gaussian Splatting for High-Quality Rendering

  • 基于距离的对齐策略,区分紧密贴合与松散分布的高斯点云。
  • 在mip-NeRF 360数据集上提升1.3dB PSNR,减少30%高斯点数量。
  • 适合游戏、AR/VR等需高质量网格渲染的场景重建应用。

最近,3D高斯点云因其生成高保真渲染结果的能力受到关注。然而,游戏、动画和AR/VR等应用普遍采用网格表示来构建和渲染三维场景。本文提出一种新方法,将网格表示与3D高斯点云融合,以实现真实世界场景的高质量渲染。我们引入基于距离的高斯点云对齐技术,使点云紧贴网格表面,并移除对渲染无贡献的冗余点云。通过计算每个高斯点云与网格表面的距离,区分紧密绑定与松散绑定的点云:前者被压平并精确对齐于网格几何结构,后者用于弥补重建网格中的渲染瑕疵。我们设计了一种将高斯点云绑定至网格几何的训练策略,并综合考虑两类点云。为此,引入多种正则化技术,在训练中精准对齐紧密绑定的点云。我们在mip-NeRF 360和Deep Blending的大规模非闭合场景数据集上验证了方法的有效性。结果表明,该方法在PSNR上优于近期基于网格的神经渲染技术2dB,较基于网格的高斯点云方法提升1.3dB,尤其在室外mip-NeRF 360数据集表现突出,显著提升渲染质量。我们对两类高斯点云进行分析,相比原始3D高斯点云方法,点云数量减少30%。

原文摘要 · Abstract (English)

Recently, 3D Gaussian splatting has gained attention for its capability to generate high-fidelity rendering results. At the same time, most applications such as games, animation, and AR/VR use mesh-based representations to represent and render 3D scenes. We propose a novel approach that integrates mesh representation with 3D Gaussian splats to perform high-quality rendering of reconstructed real-world scenes. In particular, we introduce a distance-based Gaussian splatting technique to align the Gaussian splats with the mesh surface and remove redundant Gaussian splats that do not contribute to the rendering. We consider the distance between each Gaussian splat and the mesh surface to distinguish between tightly-bound and loosely-bound Gaussian splats. The tightly-bound splats are flattened and aligned well with the mesh geometry. The loosely-bound Gaussian splats are used to account for the artifacts in reconstructed 3D meshes in terms of rendering. We present a training strategy of binding Gaussian splats to the mesh geometry, and take into account both types of splats. In this context, we introduce several regularization techniques aimed at precisely aligning tightly-bound Gaussian splats with the mesh surface during the training process. We validate the effectiveness of our method on large and unbounded scene from mip-NeRF 360 and Deep Blending datasets. Our method surpasses recent mesh-based neural rendering techniques by achieving a 2dB higher PSNR, and outperforms mesh-based Gaussian splatting methods by 1.3 dB PSNR, particularly on the outdoor mip-NeRF 360 dataset, demonstrating better rendering quality. We provide analyses for each type of Gaussian splat and achieve a reduction in the number of Gaussian splats by 30% compared to the original 3D Gaussian splatting.

三维重建高斯点云网格对齐渲染优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。