arXiv:2603.20307cs.CVcs.AI2026-03

用逐帧控制实现流畅逼真的语音驱动人脸生成,支持实时交互。

EARTalking: End-to-end GPT-style Autoregressive Talking Head Synthesis with Frame-wise Control

论文配图:EARTalking: End-to-end GPT-style Autoregressive Talking Head Synthesis with Frame-wise Control
图 1 · 摘自论文原文
  • 采用GPT式自回归框架,逐帧生成人脸视频。
  • 在多种评估中优于传统自回归方法,接近扩散模型效果。
  • 支持任意时刻交互控制,适合需要灵活生成的场景。

语音驱动的人脸生成旨在从静态肖像和语音中生成生动逼真的视频。现有自回归方法依赖中间面部表征,限制了表现力和真实感;而扩散方法按片段生成,缺乏细粒度控制且因全局去噪导致固有延迟。为此,我们提出EARTalking,一种新型端到端、GPT风格的自回归模型,用于交互式语音驱动人脸生成。该方法引入帧级、上下文感知、音频驱动的流式生成范式。为支持可变长度视频生成并保持身份一致性,提出Sink Frame Window Attention(SFA)机制。此外,为避免以往方法所需的复杂独立网络来处理多样控制信号,提出流式帧条件上下文注入(FCIC)方案,以流式、上下文方式高效注入多种控制信号,实现在每一帧及任意时刻的交互控制。实验表明,EARTalking优于现有自回归方法,性能接近扩散模型方法。本工作验证了上下文流式自回归控制的可行性,开启了灵活高效的生成新方向。代码将公开以保证可复现性。

原文摘要 · Abstract (English)

Audio-driven talking head generation aims to create vivid and realistic videos from a static portrait and speech. Existing AR-based methods rely on intermediate facial representations, which limit their expressiveness and realism. Meanwhile, diffusion-based methods generate clip-by-clip, lacking fine-grained control and causing inherent latency due to overall denoising across the window. To address these limitations, we propose EARTalking, a novel end-to-end, GPT-style autoregressive model for interactive audio-driven talking head generation. Our method introduces a novel frame-by-frame, in-context, audio-driven streaming generation paradigm. For inherently supporting variable-length video generation with identity consistency, we propose the Sink Frame Window Attention (SFA) mechanism. Furthermore, to avoid the complex, separate networks that prior works required for diverse control signals, we propose a streaming Frame Condition In-Context (FCIC) scheme. This scheme efficiently injects diverse control signals in a streaming, in-context manner, enabling interactive control at every frame and at arbitrary moments. Experiments demonstrate that EARTalking outperforms existing autoregressive methods and achieves performance comparable to diffusion-based methods. Our work demonstrates the feasibility of in-context streaming autoregressive control, unlocking a scalable direction for flexible, efficient generation. The code will be released for reproducibility.

人脸生成自回归语音驱动实时控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。