通过情绪感知扩散模型，实现精细可控的3D虚拟人表情编辑。
EmoDiffTalk:Emotion-aware Diffusion for Editable 3D Gaussian Talking Head
- 引入基于动作单元的跨模态情绪扩散机制，支持细粒度表情生成。
- 在EmoTalk3D和RenderMe-360数据集上实现更自然的唇同步与情绪细节。
- 适合需要多模态控制、连续情感编辑的虚拟人应用开发者。
当前基于3D高斯点云的逼真虚拟人说话头仍存在情绪表达操控不足的问题，尤其在多模态控制下难以实现细粒度与广泛动态的情绪编辑。本文提出可编辑3D高斯点云说话头框架EmoDiffTalk，核心是引入一种情绪感知高斯扩散机制：包含动作单元（AU）提示的高斯扩散过程以实现细粒度面部动画，并设计精准的文本到AU情绪控制器，支持通过文本输入实现准确且丰富的动态情绪编辑。在公开的EmoTalk3D和RenderMe-360数据集上的实验表明，相较于已有方法，EmoDiffTalk在情绪细腻度、唇音同步精度和可控性方面均有显著提升，为高质量、扩散驱动、多模态可编辑3D说话头合成提供了系统性路径。据我们所知，EmoDiffTalk是首个支持基于动作单元表达空间内连续、多模态情绪编辑的3D高斯点云说话头生成框架之一。
原文摘要 · Abstract (English)
Recent photo-realistic 3D talking head via 3D Gaussian Splatting still has significant shortcoming in emotional expression manipulation, especially for fine-grained and expansive dynamics emotional editing using multi-modal control. This paper introduces a new editable 3D Gaussian talking head, i.e. EmoDiffTalk. Our key idea is a novel Emotion-aware Gaussian Diffusion, which includes an action unit (AU) prompt Gaussian diffusion process for fine-grained facial animator, and moreover an accurate text-to-AU emotion controller to provide accurate and expansive dynamic emotional editing using text input. Experiments on public EmoTalk3D and RenderMe-360 datasets demonstrate superior emotional subtlety, lip-sync fidelity, and controllability of our EmoDiffTalk over previous works, establishing a principled pathway toward high-quality, diffusion-driven, multimodal editable 3D talking-head synthesis. To our best knowledge, our EmoDiffTalk is one of the first few 3D Gaussian Splatting talking-head generation framework, especially supporting continuous, multimodal emotional editing within the AU-based expression space.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。