实时语音驱动人脸动画,用自回归模型生成自然微动。
Teller: Real-Time Streaming Audio-Driven Portrait Animation with Autoregressive Motion Generation
- 采用自回归Transformer分步生成面部动作,支持流式处理。
- 0.92秒生成1秒视频,实测达25帧/秒,速度超扩散模型20倍。
- 新增时序模块增强颈部、耳饰等细节真实感,适合虚拟主播等场景。
本文提出首个基于自回归生成的实时语音驱动人脸动画框架Teller,解决传统方法生成耗时长、肢体动作不自然的问题。Teller将面部与身体动作分解为两部分:基于自回归Transformer的面部运动隐变量生成(FMLG),及利用高效时序模块(ETM)进行运动真实性优化。FMLG通过残差量化模型将关键点驱动的隐变量映射为离散动作标记,并结合音频嵌入实现端到端流式建模;ETM捕捉细微动作细节,确保颈部肌肉、耳环等部件的物理一致性。Teller推理速度极快,单秒视频生成仅需0.92秒,远超扩散模型(如Hallo的20.93秒),支持最高25 FPS实时流输出。大量实验表明,本方法在微动作表现上显著优于现有模型,人类评估中质量与真实感优势明显。
原文摘要 · Abstract (English)
In this work, we introduce the first autoregressive framework for real-time, audio-driven portrait animation, a.k.a, talking head. Beyond the challenge of lengthy animation times, a critical challenge in realistic talking head generation lies in preserving the natural movement of diverse body parts. To this end, we propose Teller, the first streaming audio-driven protrait animation framework with autoregressive motion generation. Specifically, Teller first decomposes facial and body detail animation into two components: Facial Motion Latent Generation (FMLG) based on an autoregressive transfromer, and movement authenticity refinement using a Efficient Temporal Module (ETM).Concretely, FMLG employs a Residual VQ model to map the facial motion latent from the implicit keypoint-based model into discrete motion tokens, which are then temporally sliced with audio embeddings. This enables the AR tranformer to learn real-time, stream-based mappings from audio to motion. Furthermore, Teller incorporate ETM to capture finer motion details. This module ensures the physical consistency of body parts and accessories, such as neck muscles and earrings, improving the realism of these movements. Teller is designed to be efficient, surpassing the inference speed of diffusion-based models (Hallo 20.93s vs. Teller 0.92s for one second video generation), and achieves a real-time streaming performance of up to 25 FPS. Extensive experiments demonstrate that our method outperforms recent audio-driven portrait animation models, especially in small movements, as validated by human evaluations with a significant margin in quality and realism.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。