用高斯点云重建高保真人脸几何与纹理,支持任意场景直接替换资产。
Using Gaussian Splats to Create High-Fidelity Facial Geometry and Texture
- 基于高斯点云和语义分割,仅用11张图重建中性姿态人脸
- 通过约束点云到三角网格提升结构精度,实现高质量表面重建
- 生成可独立使用的视点相关纹理,兼容标准渲染管线
我们利用日益流行的三维神经表示,构建一组未标定人脸图像的统一一致解释。采用高斯点云(Gaussian Splatting)因其显式表达更易施加约束,优于NeRF。结合分割标注对齐面部语义区域,仅需11张图像即可重建中性姿态。通过将高斯点云软约束至底层三角网格,获得更结构化的点云重建,并以此引导后续微调以提高三角网格精度。最终得到的三角网格可直接用于标准图形管线。更重要的是,精确几何使高斯点云可映射至纹理空间,作为视点相关的神经纹理使用,从而在不修改其他资产或渲染管线的前提下,在任意场景中应用高保真点云。我们采用可重光照的高斯模型分离纹理与光照,获得高分辨率反照率纹理,同样适配标准图形管线。系统具备强鲁棒性,可处理不同光照条件下的异构图像进行训练。最后,我们展示了该方法在文本驱动资产创建流程中的有效性。
原文摘要 · Abstract (English)
We leverage increasingly popular three-dimensional neural representations in order to construct a unified and consistent explanation of a collection of uncalibrated images of the human face. Our approach utilizes Gaussian Splatting, since it is more explicit and thus more amenable to constraints than NeRFs. We leverage segmentation annotations to align the semantic regions of the face, facilitating the reconstruction of a neutral pose from only 11 images (as opposed to requiring a long video). We soft constrain the Gaussians to an underlying triangulated surface in order to provide a more structured Gaussian Splat reconstruction, which in turn informs subsequent perturbations to increase the accuracy of the underlying triangulated surface. The resulting triangulated surface can then be used in a standard graphics pipeline. In addition, and perhaps most impactful, we show how accurate geometry enables the Gaussian Splats to be transformed into texture space where they can be treated as a view-dependent neural texture. This allows one to use high visual fidelity Gaussian Splatting on any asset in a scene without the need to modify any other asset or any other aspect (geometry, lighting, renderer, etc.) of the graphics pipeline. We utilize a relightable Gaussian model to disentangle texture from lighting in order to obtain a delit high-resolution albedo texture that is also readily usable in a standard graphics pipeline. The flexibility of our system allows for training with disparate images, even with incompatible lighting, facilitating robust regularization. Finally, we demonstrate the efficacy of our approach by illustrating its use in a text-driven asset creation pipeline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。