通过融合网格与3D高斯表示，提升逆渲染质量并实现实时渲染。
Phys3DGS: Physically-based 3D Gaussian Splatting for Inverse Rendering
- 采用延迟渲染与网格-高斯混合表示解决表面下高斯点干扰问题。
- 在光照重演中显著优于现有3DGS方法，接近体素网格最优水平。
- 适合需要高质量实时逆渲染的应用场景，如虚拟拍摄与数字人。
我们提出两个新思路（延迟渲染与基于网格的表示）以改进基于3D高斯喷溅（3DGS）的逆渲染质量。首先指出隐藏高斯点的问题：现有方法采用体素渲染时，位于表面下的高斯点会负面影响像素颜色。为解决此问题，我们引入延迟渲染，但发现其直接应用会引发新问题。为此，我们提出一种两阶段训练方法：(1) 利用网格提取构建混合网格-3DGS表示；(2) 引入新型正则化方法更好利用网格信息。实验表明，在光照重演任务中，该方法显著优于现有3DGS基逆渲染方法；相比当前最优的体素网格基方法，本方法在渲染质量上更优且支持实时渲染。
原文摘要 · Abstract (English)
We propose two novel ideas (adoption of deferred rendering and mesh-based representation) to improve the quality of 3D Gaussian splatting (3DGS) based inverse rendering. We first report a problem incurred by hidden Gaussians, where Gaussians beneath the surface adversely affect the pixel color in the volume rendering adopted by the existing methods. In order to resolve the problem, we propose applying deferred rendering and report new problems incurred in a naive application of deferred rendering to the existing 3DGS-based inverse rendering. In an effort to improve the quality of 3DGS-based inverse rendering under deferred rendering, we propose a novel two-step training approach which (1) exploits mesh extraction and utilizes a hybrid mesh-3DGS representation and (2) applies novel regularization methods to better exploit the mesh. Our experiments show that, under relighting, the proposed method offers significantly better rendering quality than the existing 3DGS-based inverse rendering methods. Compared with the SOTA voxel grid-based inverse rendering method, it gives better rendering quality while offering real-time rendering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。