用音频驱动生成逼真会动的肖像,表情和动作更自然可控。
FantasyTalking: Realistic Talking Portrait Generation via Coherent Motion Synthesis
- 分两阶段对齐音视频:全局运动靠片段级对齐,唇动靠帧级掩码精调。
- 生成视频在真实感、连贯性、身份保持上优于现有方法,可调节表情强度。
- 适合影视特效、虚拟主播等需要高保真动态肖像的场景。
从单张静态肖像生成逼真可动画的虚拟人物仍具挑战。现有方法难以捕捉细微面部表情、整体身体动作及动态背景。为此,我们提出一种新框架,利用预训练视频扩散变换模型生成高质量、连贯的说话肖像,并支持运动动态控制。核心是双阶段音视频对齐策略:第一阶段采用片段级训练,对齐整个场景(包括参考肖像、上下文物体与背景)的音频驱动动态;第二阶段通过唇部追踪掩码在帧级别精修口型,确保与音频精确同步。为在保持身份一致性的同时提升动作灵活性,我们用面向面部的交叉注意力模块替代传统参考网络。此外,引入运动强度调制模块,显式控制表情与身体动作强度,实现超越单纯口型动作的可控表达。大量实验表明,本方法在真实感、连贯性、运动强度与身份保持方面均优于现有技术。
原文摘要 · Abstract (English)
Creating a realistic animatable avatar from a single static portrait remains challenging. Existing approaches often struggle to capture subtle facial expressions, the associated global body movements, and the dynamic background. To address these limitations, we propose a novel framework that leverages a pretrained video diffusion transformer model to generate high-fidelity, coherent talking portraits with controllable motion dynamics. At the core of our work is a dual-stage audio-visual alignment strategy. In the first stage, we employ a clip-level training scheme to establish coherent global motion by aligning audio-driven dynamics across the entire scene, including the reference portrait, contextual objects, and background. In the second stage, we refine lip movements at the frame level using a lip-tracing mask, ensuring precise synchronization with audio signals. To preserve identity without compromising motion flexibility, we replace the commonly used reference network with a facial-focused cross-attention module that effectively maintains facial consistency throughout the video. Furthermore, we integrate a motion intensity modulation module that explicitly controls expression and body motion intensity, enabling controllable manipulation of portrait movements beyond mere lip motion. Extensive experimental results show that our proposed approach achieves higher quality with better realism, coherence, motion intensity, and identity preservation. Ours project page: https://fantasy-amap.github.io/fantasy-talking/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。