用野外照片训练出可控制表情和风格的3D人脸模型,支持实时渲染。
StyleMorpheus: A Style-Based 3D-Aware Morphable Face Model
- 用自编码结构学习解耦的参数化代码空间,无需精确3D重建
- 在野外图像上训练,实现比现有方法更优的3D感知重建效果
- 支持风格混合、颜色编辑,适合虚拟现实与人脸编辑应用
在3D人脸建模领域,近期的3D感知神经渲染方法能以任意视角生成逼真人脸图像。然而,参数化可控的3D感知人脸模型训练仍依赖大规模实验室采集数据。为此,本文提出首个基于风格的神经3D可变形人脸模型(StyleMorpheus),在野外图像上进行训练。该模型继承3DMM的解耦控制能力(身份、表情、外观),但无需精确重建显式3D形状。StyleMorpheus采用自编码器结构:编码器学习代表性解耦参数码空间,解码器通过网络不同模块中的形状与外观相关风格码增强解耦性。此外,通过基于风格的生成对抗学习对解码器进行微调,实现逼真的3D渲染质量。该风格化设计使模型达到当前最优的3D感知人脸重建性能,同时支持解耦控制。模型具备实时渲染速度,适用于虚拟现实应用。我们还展示了其在风格混合与色彩编辑等人脸编辑任务中的能力。
原文摘要 · Abstract (English)
For 3D face modeling, the recently developed 3D-aware neural rendering methods are able to render photorealistic face images with arbitrary viewing directions. The training of the parametric controllable 3D-aware face models, however, still relies on a large-scale dataset that is lab-collected. To address this issue, this paper introduces "StyleMorpheus", the first style-based neural 3D Morphable Face Model (3DMM) that is trained on in-the-wild images. It inherits 3DMM's disentangled controllability (over face identity, expression, and appearance) but without the need for accurately reconstructed explicit 3D shapes. StyleMorpheus employs an auto-encoder structure. The encoder aims at learning a representative disentangled parametric code space and the decoder improves the disentanglement using shape and appearance-related style codes in the different sub-modules of the network. Furthermore, we fine-tune the decoder through style-based generative adversarial learning to achieve photorealistic 3D rendering quality. The proposed style-based design enables StyleMorpheus to achieve state-of-the-art 3D-aware face reconstruction results, while also allowing disentangled control of the reconstructed face. Our model achieves real-time rendering speed, allowing its use in virtual reality applications. We also demonstrate the capability of the proposed style-based design in face editing applications such as style mixing and color editing. Project homepage: https://github.com/ubc-3d-vision-lab/StyleMorpheus.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。