通过捕捉音频驱动的面部动态差异,实现人脸动画加速且保持音画同步。
SyncCache: Exploiting Asymmetric Dynamics for Fast Audio-Driven Portrait Animation

- 按区域优先缓存动态人脸部分,忽略静态背景
- 缓存稳定残差,仅重算音频相关部分,提速3.75倍以上
- 无需训练,适合实时人脸动画生成场景
扩散变换器(DiTs)显著提升了音频驱动的人脸动画效果,但其高计算开销导致推理延迟严重。尽管无训练扩散缓存可显著加速,现有方法主要面向文本生成,忽视了音频驱动人脸动画中的空间与模态不平衡问题。本文提出SyncCache,一种针对DiT人脸动画的无训练缓存加速方法,显式利用非对称动态特性。具体而言,由音频驱动的高频动态集中在人脸区域,更具挑战性且更需缓存复用,而低频视觉背景则相对稳定。首先,引入空间非对称探测,优先关注动态人脸区域的误差敏感度;其次,通过模态解耦缓存,跳过复杂扩散块,复用稳定跨块残差,同时仅重新计算轻量级音频块以保证精确唇同步。此外,引入缓存比例控制缓存容量,并将内存自适应缓存选择建模为离线动态规划问题,无在线开销。大量实验表明,SyncCache在HunyuanVideo-Avatar和Wan-S2V上分别实现最高4.12倍和3.75倍加速,视觉保真度近乎无损,且音频对齐精准。
原文摘要 · Abstract (English)
Diffusion Transformers (DiTs) have significantly advanced audio-driven portrait animation, but their high computational cost leads to substantial inference latency. Although training-free diffusion caching accelerates inference significant, existing methods are primarily developed for text-conditioned generation and overlook the spatial and modality imbalances inherent in audio-driven portrait animation. In this paper, we propose SyncCache, a training-free caching acceleration method tailored for DiT-based portrait animation that explicitly exploits asymmetric dynamics. Specifically, high-frequency dynamics driven by audio conditions and concentrated in human regions are more challenging and critical to cache and reuse than the low-frequency visual background in portrait animation. First, we introduce Spatially-Asymmetric Probing to prioritize error sensitivity in dynamic human region. Second, through Modality-Decoupled Caching, we bypass heavy DiT block by reusing stable inter-block residuals, while continuously recomputing lightweight audio blocks to preserve precise lip synchronization. Furthermore, we introduce a cache ratio to control cache capacity and formulate memory-adaptive cache selection as an offline dynamic programming problem without online overhead. Extensive experiments demonstrate that SyncCache achieves superior speed-quality trade-offs, delivering up to 4.12x acceleration on HunyuanVideo-Avatar and 3.75x on Wan-S2V with near-lossless visual fidelity and precise audio alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。