用一张图生成稳定说话人脸，还能自定义表情
MAGIC-Talk: Motion-aware Audio-Driven Talking Face Generation with Customizable Identity Control
- 基于扩散模型，单图保身份，支持文本控制微调
- 引入结构化运动先验，长视频无抖动、不糊脸
- 适合虚拟主播、数字人定制，无需调参
音频驱动的说话人脸生成在数字媒体和虚拟化身中备受关注。尽管近期方法提升了音唇同步效果，但在长时间生成中仍面临时序不一致、身份保持困难与可定制性差的问题。为此，我们提出 MAGIC-Talk，一种基于扩散模型的一次性可定制、时序稳定的说话人脸生成框架。MAGIC-Talk 包含 ReferenceNet，用于保留身份并支持文本提示进行细粒度面部编辑；以及 AnimateNet，通过结构化运动先验增强动作连贯性。与需要多参考图或微调的方法不同，MAGIC-Talk 仅需一张参考图像即可维持身份一致性，并实现帧间平滑过渡。此外，采用渐进式潜在融合策略，有效降低长视频中的运动不一致和闪烁问题。大量实验表明，MAGIC-Talk 在视觉质量、身份保留和同步准确率上均优于现有最佳方法，为说话人脸生成提供了鲁棒解决方案。
原文摘要 · Abstract (English)
Audio-driven talking face generation has gained significant attention for applications in digital media and virtual avatars. While recent methods improve audio-lip synchronization, they often struggle with temporal consistency, identity preservation, and customization, especially in long video generation. To address these issues, we propose MAGIC-Talk, a one-shot diffusion-based framework for customizable and temporally stable talking face generation. MAGIC-Talk consists of ReferenceNet, which preserves identity and enables fine-grained facial editing via text prompts, and AnimateNet, which enhances motion coherence using structured motion priors. Unlike previous methods requiring multiple reference images or fine-tuning, MAGIC-Talk maintains identity from a single image while ensuring smooth transitions across frames. Additionally, a progressive latent fusion strategy is introduced to improve long-form video quality by reducing motion inconsistencies and flickering. Extensive experiments demonstrate that MAGIC-Talk outperforms state-of-the-art methods in visual quality, identity preservation, and synchronization accuracy, offering a robust solution for talking face generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。