在遮挡情况下重建多样且逼真的3D人脸,解决单图重建的模糊性问题。
OFER: Occluded Face Expression Reconstruction
- 用两个扩散模型分别生成人脸形状和表情参数,捕捉多解可能性
- 新提出排序机制,基于形状准确度分数选出最匹配的形状结果
- 构建新数据集CO-545,验证遮挡下表情重建性能,适合表情生成研究者
从单张图像重建3D人脸是典型的病态问题,尤其在存在遮挡时挑战更大。由于观测信息减少且遮挡引入额外歧义,同一输入可能对应多个合理重建结果。尽管该问题普遍存在,现有方法极少考虑其多假设特性。本文提出OFER,一种新颖的单图像3D人脸重建方法,可在强遮挡条件下生成合理、多样且具表现力的3D人脸。具体而言,我们训练两个扩散模型,分别根据输入图像生成人脸参数模型的形状与表情系数,从而捕获问题的多模态特性,输出一组解分布。为确保不同表情间的一致性,关键在于选择最优匹配的形状。为此,我们设计了一种新颖的排序机制,依据预测的形状准确度分数对形状扩散网络输出进行排序。我们在标准基准上评估方法,并引入新协议与数据集CO-545,用于评估遮挡条件下表达性人脸的重建精度。实验表明,本方法在遮挡场景下优于现有方法,同时可为同一图像生成多样化表情。
原文摘要 · Abstract (English)
Reconstructing 3D face models from a single image is an inherently ill-posed problem, which becomes even more challenging in the presence of occlusions. In addition to fewer available observations, occlusions introduce an extra source of ambiguity where multiple reconstructions can be equally valid. Despite the ubiquity of the problem, very few methods address its multi-hypothesis nature. In this paper we introduce OFER, a novel approach for single-image 3D face reconstruction that can generate plausible, diverse, and expressive 3D faces, even under strong occlusions. Specifically, we train two diffusion models to generate the shape and expression coefficients of a face parametric model, conditioned on the input image. This approach captures the multi-modal nature of the problem, generating a distribution of solutions as output. However, to maintain consistency across diverse expressions, the challenge is to select the best matching shape. To achieve this, we propose a novel ranking mechanism that sorts the outputs of the shape diffusion network based on predicted shape accuracy scores. We evaluate our method using standard benchmarks and introduce CO-545, a new protocol and dataset designed to assess the accuracy of expressive faces under occlusion. Our results show improved performance over occlusion-based methods, while also enabling the generation of diverse expressions for a given image.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。