用2D高斯模拟头像，实现无需3D标签的实时人脸重建。
SHeaP: Self-Supervised Head Geometry Predictor Learned via 2D Gaussians
- 用可驱动的高斯点云替代传统网格，实现更高效的自监督训练。
- 在NoW和新表情数据集上，几何精度超越现有自监督方法。
- 生成的表情细节丰富，情绪识别效果优于当前最优模型。
从单目图像和视频中实现准确、实时的人脸3D重建，是众多视觉应用的基础。由于大规模3D真实数据难以获取，以往方法尝试通过大量2D视频进行自监督学习。通常采用可微分网格渲染，虽有效但存在局限。为此，本文提出SHeaP（基于2D高斯的自监督头像几何预测器）。给定源图像，模型预测3DMM网格与绑定至该网格的一组高斯点。随后将该绑定头像重动画以匹配目标帧，并通过光度损失反向传播至3DMM和高斯预测网络。实验表明，使用高斯点云渲染显著提升了自监督学习效果。仅在2D数据上训练，本方法在NoW基准（中性表情）及新构建的非中性表情基准上均超越现有自监督方法。同时生成的网格表达力强，在情绪分类任务中达到业界领先水平。
原文摘要 · Abstract (English)
Accurate, real-time 3D reconstruction of human heads from monocular images and videos underlies numerous visual applications. As 3D ground truth data is hard to come by at scale, previous methods have sought to learn from abundant 2D videos in a self-supervised manner. Typically, this involves the use of differentiable mesh rendering, which is effective but faces limitations. To improve on this, we propose SHeaP (Self-supervised Head Geometry Predictor Learned via 2D Gaussians). Given a source image, we predict a 3DMM mesh and a set of Gaussians that are rigged to this mesh. We then reanimate this rigged head avatar to match a target frame, and backpropagate photometric losses to both the 3DMM and Gaussian prediction networks. We find that using Gaussians for rendering substantially improves the effectiveness of this self-supervised approach. Training solely on 2D data, our method surpasses existing self-supervised approaches in geometric evaluations on the NoW benchmark for neutral faces and a new benchmark for non-neutral expressions. Our method also produces highly expressive meshes, outperforming state-of-the-art in emotion classification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。