让语音驱动的3D人脸动画更自然可编辑，支持多样动作与精细控制。
3DiFACE: Synthesizing and Editing Holistic 3D Facial Animation
- 用全卷积扩散模型生成多种合理口型和头部动作
- 单个音频输入可生成多组不同表情与动作组合
- 支持关键帧编辑和插值，适合影视动画与虚拟人开发
当前语音驱动的3D人脸动画在个性化生成与精确控制方面仍面临挑战，尤其在动画编辑上需依赖专业人员且耗时。现有方法多聚焦于风格或情绪控制，无法对已有动画进行局部修改或重生成，且忽略同一语音输入存在多种合理口型与头部运动的可能性。为此，我们提出3DiFACE，一种端到端的语音驱动3D人脸动画生成与编辑方法。该方法通过一个全卷积扩散模型利用训练数据中的音素级多样性，结合说话风格个性化与稀疏引导运动扩散机制，实现对口型与头部动作的精准控制。实验表明，3DiFACE可在单一音频输入下生成多样化、高保真的人脸动画，并支持基于关键帧和插值的灵活编辑。代码与模型已公开。
原文摘要 · Abstract (English)
Creating personalized 3D animations with precise control and realistic head motions remains challenging for current speech-driven 3D facial animation methods. Editing these animations is especially complex and time consuming, requires precise control and typically handled by highly skilled animators. Most existing works focus on controlling style or emotion of the synthesized animation and cannot edit/regenerate parts of an input animation. They also overlook the fact that multiple plausible lip and head movements can match the same audio input. To address these challenges, we present 3DiFACE, a novel method for holistic speech-driven 3D facial animation. Our approach produces diverse plausible lip and head motions for a single audio input and allows for editing via keyframing and interpolation. Specifically, we propose a fully-convolutional diffusion model that can leverage the viseme-level diversity in our training corpus. Additionally, we employ a speaking-style personalization and a novel sparsely-guided motion diffusion to enable precise control and editing. Through quantitative and qualitative evaluations, we demonstrate that our method is capable of generating and editing diverse holistic 3D facial animations given a single audio input, with control between high fidelity and diversity. Code and models are available here: https://balamuruganthambiraja.github.io/3DiFACE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。