arXiv:2410.04221cs.CV2024-10ICLR被引 33

用语音驱动生成同步手势的高保真视频,解决动作错位和画面瑕疵问题。

TANGO: Co-Speech Gesture Video Reenactment with Hierarchical Audio Motion Embedding and Diffusion Interpolation

  • 基于分层音频动作嵌入与图结构检索,实现跨模态对齐。
  • 扩散模型生成过渡帧,保持参考视频外观一致性。
  • 适合影视动画、虚拟主播等需要精准语音-手势同步的场景。

我们提出TANGO,一个生成伴随说话动作的身体手势视频的框架。给定一段单人参考视频和目标语音音频,TANGO可生成与语音同步的高保真视频。该方法基于手势视频重演(GVR),通过有向图结构分割并检索视频片段——将视频帧视为节点,有效转移关系作为边。针对GVR的两大缺陷:语音-动作错位与GAN生成过渡帧产生的视觉伪影,我们提出:(i) 使用潜在特征距离检索手势,提升跨模态对齐;为此构建了分层联合嵌入空间AuMoCLIP,使潜在特征有效建模语音与动作的关系;(ii) 引入基于扩散模型的过渡帧生成方法,即外观一致插值(ACInterp),在AnimateAnyone基础上加入参考动作模块与仿射背景流,确保生成帧与参考视频外观一致。将上述组件整合进图检索框架,TANGO可稳定生成逼真、语音同步的视频,在生成与检索类方法中均优于现有技术。代码与预训练模型已公开: https://pantomatrix.github.io/TANGO/

原文摘要 · Abstract (English)

We present TANGO, a framework for generating co-speech body-gesture videos. Given a few-minute, single-speaker reference video and target speech audio, TANGO produces high-fidelity videos with synchronized body gestures. TANGO builds on Gesture Video Reenactment (GVR), which splits and retrieves video clips using a directed graph structure - representing video frames as nodes and valid transitions as edges. We address two key limitations of GVR: audio-motion misalignment and visual artifacts in GAN-generated transition frames. In particular, (i) we propose retrieving gestures using latent feature distance to improve cross-modal alignment. To ensure the latent features could effectively model the relationship between speech audio and gesture motion, we implement a hierarchical joint embedding space (AuMoCLIP); (ii) we introduce the diffusion-based model to generate high-quality transition frames. Our diffusion model, Appearance Consistent Interpolation (ACInterp), is built upon AnimateAnyone and includes a reference motion module and homography background flow to preserve appearance consistency between generated and reference videos. By integrating these components into the graph-based retrieval framework, TANGO reliably produces realistic, audio-synchronized videos and outperforms all existing generative and retrieval methods. Our codes and pretrained models are available: \url{https://pantomatrix.github.io/TANGO/}

视频重演语音驱动扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。