用一张照片给3D虚拟人换妆,还能保持表情一致、身份不变。
AvatarMakeup: Realistic Makeup Transfer for 3D Animatable Head Avatars
- 先粗后精:用扩散模型生成妆容图,再通过全局UV图确保多视角一致性。
- 在动画中保持妆容稳定,人脸身份不丢失,细节可控。
- 适合需要高保真虚拟人形象的元宇宙、游戏开发者。
与真实人脸美化类似,3D虚拟角色也需要个性化定制以提升视觉吸引力,但该领域仍研究不足。现有3D高斯编辑方法虽可应用于面部化妆,但难以满足真实妆容的核心要求:1)驱动表情时外观保持一致;2)化妆过程中保留身份特征;3)精细控制细节。为此,我们提出专用3D化妆方法AvatarMakeup,利用预训练扩散模型从单张参考照片中迁移妆容。采用粗到精策略,首先保证外观和身份一致性,再细化细节。具体地,扩散模型生成妆容图像作为监督信号。由于扩散过程存在不确定性,生成图像在不同视角和表情下不一致。因此,我们提出Coherent Duplication方法,通过优化全局UV图(基于生成妆容图像的平均面部属性),实现目标的粗略化妆,并确保动态与多视角的一致性。通过查询全局UV图,可从任意视角和表情生成一致的化妆引导,优化目标角色。在粗妆基础上,进一步引入Refinement Module融合扩散模型,提升妆容质量。实验表明,AvatarMakeup在动画过程中实现了最先进的妆容转移质量与一致性。
原文摘要 · Abstract (English)
Similar to facial beautification in real life, 3D virtual avatars require personalized customization to enhance their visual appeal, yet this area remains insufficiently explored. Although current 3D Gaussian editing methods can be adapted for facial makeup purposes, these methods fail to meet the fundamental requirements for achieving realistic makeup effects: 1) ensuring a consistent appearance during drivable expressions, 2) preserving the identity throughout the makeup process, and 3) enabling precise control over fine details. To address these, we propose a specialized 3D makeup method named AvatarMakeup, leveraging a pretrained diffusion model to transfer makeup patterns from a single reference photo of any individual. We adopt a coarse-to-fine idea to first maintain the consistent appearance and identity, and then to refine the details. In particular, the diffusion model is employed to generate makeup images as supervision. Due to the uncertainties in diffusion process, the generated images are inconsistent across different viewpoints and expressions. Therefore, we propose a Coherent Duplication method to coarsely apply makeup to the target while ensuring consistency across dynamic and multiview effects. Coherent Duplication optimizes a global UV map by recoding the averaged facial attributes among the generated makeup images. By querying the global UV map, it easily synthesizes coherent makeup guidance from arbitrary views and expressions to optimize the target avatar. Given the coarse makeup avatar, we further enhance the makeup by incorporating a Refinement Module into the diffusion model to achieve high makeup quality. Experiments demonstrate that AvatarMakeup achieves state-of-the-art makeup transfer quality and consistency throughout animation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。