用语音引导的自回归模型,解决说话头视频的帧间闪烁问题。
FluentAvatar: Flicker-Free Talking-Head Animation via Phoneme-Guided Autoregressive Modeling
- 分两阶段生成:先对齐音素生成关键帧,再自适应插值过渡帧。
- 在CMLR和HDTF数据集上,帧间闪烁接近真实水平,FVD表现最佳。
- 提出背景隔离评估指标BG-Flicker,更准确衡量视觉稳定性。
当前说话头生成已从GAN转向扩散模型,虽在视觉保真度和时序一致性上取得进展,但帧间闪烁仍普遍。根本原因在于随机初始化导致去噪轨迹变化,引发相邻帧间的短时突变。我们通过固定输入、改变随机种子的对照实验验证:不同采样间闪烁模式差异显著,平均种子间皮尔逊相关系数仅r=0.15。为此,我们提出FluentAvatar,一种基于音素表示的两阶段自回归框架。第一阶段通过音素-帧因果注意力掩码生成音素对齐的关键帧;第二阶段采用时间戳感知的自适应策略,结合选择性状态空间建模合成过渡帧。此外,我们引入BG-Flicker——一种仅关注背景的闪烁评估指标,提升评估可靠性。在CMLR和HDTF数据集上的实验表明,FluentAvatar在视觉保真度、口型同步与时序稳定性方面表现优异,两项数据集的FVD均达最优,且BG-Flicker结果接近真实水平。代码、模型与界面将开源,促进后续研究。
原文摘要 · Abstract (English)
Current talking-head generation has gradually shifted from GAN-based methods to diffusion-based paradigms, achieving remarkable progress in visual fidelity and temporal consistency. However, inter-frame flicker remains prevalent in existing diffusion-based methods. An important reason is that denoising trajectory variation induced by stochastic initialization leaves residual inter-frame inconsistencies, which manifest as short-term, abrupt visual fluctuations between adjacent frames. To further verify this, we conduct a controlled study by fixing the input while varying only the random seed. The results show markedly different flicker patterns across samplings, with a mean inter-seed Pearson correlation of only r = 0.15. This motivates us to explore autoregressive generation, which models frames sequentially and provides a more direct prior for temporal continuity. Based on this, we propose FluentAvatar, a two-stage autoregressive framework built on phoneme representations. First, Facial Keyframe Generation produces phoneme-aligned keyframes under a Phoneme-Frame Causal Attention Mask, and Inter-frame Interpolation synthesizes transition frames via a timestamp-aware adaptive strategy built upon selective state space modeling. Moreover, we introduce BG-Flicker, a background-isolated metric for talking-head videos that enables more reliable evaluation of inter-frame flicker. Experiments on CMLR and HDTF demonstrate that FluentAvatar achieves strong performance in visual fidelity, lip synchronization, and temporal stability, attaining the best FVD on both datasets and BG-Flicker results close to ground truth. The code, the model, and the interface will be released to facilitate further research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。