arXiv:2504.02433cs.CV2025-04NeurIPS被引 12

一句话生成带表情的说话视频,还能模仿目标人物风格。

OmniTalker: One-shot Real-time Text-Driven Talking Audio-Video Generation With Multimodal Style Mimicking

  • 用双分支扩散模型同步生成语音和视频,浅层跨模态融合,深层独立处理。
  • 25帧/秒实时运行,一次输入即可还原说话人声音与面部动作风格。
  • 无需额外提取风格,适合影视创作、虚拟主播等实时应用。

尽管音频驱动的虚拟说话人生成已取得显著进展,但文本驱动方法仍研究不足。本文提出OmniTalker,一个统一框架,能从输入文本联合生成同步的语音与视频内容,并模仿目标身份的说话风格,包括语音特征、头部运动和面部动态。框架采用双分支扩散Transformer(DiT)结构,分别负责音频生成与视频合成;浅层引入跨模态融合模块,深层则独立处理各模态,音频经声码器解码,视频由基于GAN的高质量渲染器生成。通过掩码填充策略利用DiT的上下文学习能力,模型可在无需显式风格提取模块的情况下同时捕捉音视频风格。得益于DiT骨干网络的高效性与优化的视觉渲染器,OmniTalker实现25 FPS的实时推理。据我们所知,OmniTalker是首个能在单次输入下实时联合建模语音与面部风格的框架。大量实验表明,其在生成质量上优于现有方法,尤其在保持风格一致性与精准音视频同步方面表现突出,同时具备高效推理能力。

原文摘要 · Abstract (English)

Although significant progress has been made in audio-driven talking head generation, text-driven methods remain underexplored. In this work, we present OmniTalker, a unified framework that jointly generates synchronized talking audio-video content from input text while emulating the speaking and facial movement styles of the target identity, including speech characteristics, head motion, and facial dynamics. Our framework adopts a dual-branch diffusion transformer (DiT) architecture, with one branch dedicated to audio generation and the other to video synthesis. At the shallow layers, cross-modal fusion modules are introduced to integrate information between the two modalities. In deeper layers, each modality is processed independently, with the generated audio decoded by a vocoder and the video rendered using a GAN-based high-quality visual renderer. Leveraging the in-context learning capability of DiT through a masked-infilling strategy, our model can simultaneously capture both audio and visual styles without requiring explicit style extraction modules. Thanks to the efficiency of the DiT backbone and the optimized visual renderer, OmniTalker achieves real-time inference at 25 FPS. To the best of our knowledge, OmniTalker is the first one-shot framework capable of jointly modeling speech and facial styles in real time. Extensive experiments demonstrate its superiority over existing methods in terms of generation quality, particularly in preserving style consistency and ensuring precise audio-video synchronization, all while maintaining efficient inference.

语音视频生成扩散模型实时生成风格模仿

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。