arXiv:2508.03457cs.GRcs.CV2025-08AAAI被引 2

提出READ框架,实现音频驱动人脸生成的实时高效与稳定。

READ: Real-time and Efficient Asynchronous Diffusion for Audio-driven Talking Head Generation

  • 用时空压缩潜空间降低计算量,加速生成过程。
  • 通过语音自编码器对齐音视频潜码,提升同步性。
  • 创新异步噪声调度,保障长时生成的一致性与速度。

扩散模型虽推动了音频驱动人脸生成的发展,但推理速度极慢限制了实际应用。本文提出READ,一种基于扩散-变换器的实时人脸生成框架。首先通过时间变分自编码器(temporal VAE)学习时空高度压缩的视频潜空间,显著减少令牌数量以加速生成。为在该压缩潜空间内实现更好的音视频对齐,提出预训练语音自编码器(SpeechAE),生成对应视频潜空间的时间压缩语音潜码。这些潜表示由精心设计的音频到视频扩散变换器(A2V-DiT)建模,实现高效人脸合成。此外,为确保长时间生成中的时序一致性与加速推理,提出新型异步噪声调度器(ANS),在训练和推理中均采用异步加噪与运动引导生成策略,保证生成视频片段的一致性。实验表明,READ在生成质量上优于现有方法,显著降低运行时间,在质量和速度间取得最优平衡,并在长时间生成中保持度量稳定。

原文摘要 · Abstract (English)

The introduction of diffusion models has brought significant advances to the field of audio-driven talking head generation. However, the extremely slow inference speed severely limits the practical implementation of diffusion-based talking head generation models. In this study, we propose READ, a real-time diffusion-transformer-based talking head generation framework. Our approach first learns a spatiotemporal highly compressed video latent space via a temporal VAE, significantly reducing the token count to accelerate generation. To achieve better audio-visual alignment within this compressed latent space, a pre-trained Speech Autoencoder (SpeechAE) is proposed to generate temporally compressed speech latent codes corresponding to the video latent space. These latent representations are then modeled by a carefully designed Audio-to-Video Diffusion Transformer (A2V-DiT) backbone for efficient talking head synthesis. Furthermore, to ensure temporal consistency and accelerated inference in extended generation, we propose a novel asynchronous noise scheduler (ANS) for both the training and inference processes of our framework. The ANS leverages asynchronous add-noise and asynchronous motion-guided generation in the latent space, ensuring consistency in generated video clips. Experimental results demonstrate that READ outperforms state-of-the-art methods by generating competitive talking head videos with significantly reduced runtime, achieving an optimal balance between quality and speed while maintaining robust metric stability in long-time generation.

人脸生成扩散模型实时推理音视频对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。