通过缓存与并行去噪加速语音驱动人脸生成，速度提升显著且画质不变。
Lightning Fast Caching-based Parallel Denoising Prediction for Accelerating Talking Head Generation
- 缓存静态特征跳过大部分模型层，支持并行去噪预测。
- 推理速度提升10倍以上，视频质量保持接近原模型水平。
- 适合需要实时生成高质量人脸视频的应用场景。
基于扩散模型的语音驱动人脸生成可生成高保真、逼真的视频，但推理速度慢，限制了实际应用。现有通用扩散模型加速方法无法利用语音驱动人脸生成特有的时空冗余性。本文提出一种任务专用框架，通过两项关键创新解决效率问题：首先，提出闪电式缓存并行去噪预测（LightningCP），缓存静态特征以在推理时跳过大部分模型层，并利用缓存特征与估计噪声潜变量作为输入实现并行预测，高效绕过串行采样过程；其次，提出解耦前景注意力（DFA），利用人脸视频中空间区域的解耦特性，将注意力限制在动态前景区域，进一步加速注意力计算；此外，部分层移除参考特征以获得额外提速。大量实验表明，该框架显著提升推理速度，同时保持视频质量。
原文摘要 · Abstract (English)
Diffusion-based talking head models generate high-quality, photorealistic videos but suffer from slow inference, limiting practical applications. Existing acceleration methods for general diffusion models fail to exploit the temporal and spatial redundancies unique to talking head generation. In this paper, we propose a task-specific framework addressing these inefficiencies through two key innovations. First, we introduce Lightning-fast Caching-based Parallel denoising prediction (LightningCP), caching static features to bypass most model layers in inference time. We also enable parallel prediction using cached features and estimated noisy latents as inputs, efficiently bypassing sequential sampling. Second, we propose Decoupled Foreground Attention (DFA) to further accelerate attention computations, exploiting the spatial decoupling in talking head videos to restrict attention to dynamic foreground regions. Additionally, we remove reference features in certain layers to bring extra speedup. Extensive experiments demonstrate that our framework significantly improves inference speed while preserving video quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。