arXiv:2410.07757cs.CV2024-10中稿 · ACMMM 2024被引 32

构建首个多模态3D人脸动画数据集,支持文本驱动精细表情生成。

MMHead: Towards Fine-grained Multi-modal 3D Facial Animation

  • 基于单目视频自动重建3D面部动作,结合AU检测与ChatGPT生成分层文本标注。
  • 提出MM2Face模型,在文本引导下生成多样且逼真的3D面部运动,表现优异。
  • 适用于视频生成、虚拟人交互等场景,推动多模态3D动画发展。

3D人脸动画在多媒体领域应用广泛,音频驱动已取得良好成果。但文本引导的多模态3D人脸动画因缺乏数据而研究较少。为此,本文构建了大规模多模态3D人脸动画数据集MMHead,包含49小时3D面部动作序列、语音音频及多层次文本注释。每条文本涵盖抽象动作与情绪描述、细粒度面部与头部运动(如表情和头姿)描述,以及可能引发该情绪的三种情境。通过整合五个公开2D人像视频数据集,提出自动化流程:1)从单目视频重建3D面部动作序列;2)借助AU检测与ChatGPT生成分层文本标注。基于MMHead,建立两项新任务基准:文本诱导3D说话头动画与文本到3D面部动作生成。同时提出轻量高效的VQ-VAE方法MM2Face,统一多模态信息,生成多样且合理3D面部动作,在两个基准上均达到领先性能。大量实验与分析验证了该数据集与基准在推动多模态3D人脸动画发展上的巨大潜力。

原文摘要 · Abstract (English)

3D facial animation has attracted considerable attention due to its extensive applications in the multimedia field. Audio-driven 3D facial animation has been widely explored with promising results. However, multi-modal 3D facial animation, especially text-guided 3D facial animation is rarely explored due to the lack of multi-modal 3D facial animation dataset. To fill this gap, we first construct a large-scale multi-modal 3D facial animation dataset, MMHead, which consists of 49 hours of 3D facial motion sequences, speech audios, and rich hierarchical text annotations. Each text annotation contains abstract action and emotion descriptions, fine-grained facial and head movements (i.e., expression and head pose) descriptions, and three possible scenarios that may cause such emotion. Concretely, we integrate five public 2D portrait video datasets, and propose an automatic pipeline to 1) reconstruct 3D facial motion sequences from monocular videos; and 2) obtain hierarchical text annotations with the help of AU detection and ChatGPT. Based on the MMHead dataset, we establish benchmarks for two new tasks: text-induced 3D talking head animation and text-to-3D facial motion generation. Moreover, a simple but efficient VQ-VAE-based method named MM2Face is proposed to unify the multi-modal information and generate diverse and plausible 3D facial motions, which achieves competitive results on both benchmarks. Extensive experiments and comprehensive analysis demonstrate the significant potential of our dataset and benchmarks in promoting the development of multi-modal 3D facial animation.

3D人脸动画多模态生成文本驱动数据集

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。