用3D人脸模型实现高精度表情姿态控制，保留细节且无需微调。
ControlFace: Harnessing Facial Parametric Control for Face Rigging
- 基于双分支U-Net，分离身份特征与生成过程。
- 控制混合模块提升姿态表情控制精度，保留细微身份特征。
- 适合需要灵活控制人脸的影视动画、虚拟人应用。
人脸图像的操控（如姿态、表情、光照）即人脸绑定，是计算机视觉中的复杂任务。现有方法依赖图像数据集，需针对个体微调，难以保持精细身份与语义细节，限制了实用性。为此，我们提出ControlFace，一种基于3DMM渲染条件的新型人脸绑定方法，实现灵活高保真控制。采用双分支U-Net：FaceNet捕捉身份与细粒度特征，另一分支专注生成。通过控制混合模块编码目标对齐与参考对齐控制间的相关特征，并引入参考控制引导机制，引导生成过程以更好遵循控制指令。在人脸视频数据集上训练，充分挖掘FaceNet的丰富表征，同时确保控制一致性。大量实验表明，ControlFace在身份保留与控制精度方面表现更优，凸显其实用性。
原文摘要 · Abstract (English)
Manipulation of facial images to meet specific controls such as pose, expression, and lighting, also known as face rigging, is a complex task in computer vision. Existing methods are limited by their reliance on image datasets, which necessitates individual-specific fine-tuning and limits their ability to retain fine-grained identity and semantic details, reducing practical usability. To overcome these limitations, we introduce ControlFace, a novel face rigging method conditioned on 3DMM renderings that enables flexible, high-fidelity control. We employ a dual-branch U-Nets: one, referred to as FaceNet, captures identity and fine details, while the other focuses on generation. To enhance control precision, the control mixer module encodes the correlated features between the target-aligned control and reference-aligned control, and a novel guidance method, reference control guidance, steers the generation process for better control adherence. By training on a facial video dataset, we fully utilize FaceNet's rich representations while ensuring control adherence. Extensive experiments demonstrate ControlFace's superior performance in identity preservation and control precision, highlighting its practicality. Please see the project website: https://cvlab-kaist.github.io/ControlFace/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。