arXiv:2608.13602cs.MMcs.CV2026-08

实现分钟级实时音视频虚拟人生成,支持长期交互。

Omni-LiveAvatar: Minute-Level Real-Time Streaming Joint Audio-Video Avatar Generation

论文配图:Omni-LiveAvatar: Minute-Level Real-Time Streaming Joint Audio-Video Avatar Generation
图 1 · 摘自论文原文
  • 采用渐进式自回归蒸馏,将大模型转为几步生成的实时系统。
  • 在单卡上比基线快33倍,生成质量全面超越现有方法。
  • 适合需要长时间互动的虚拟主播、数字人应用。

音视频联合生成模型是构建沉浸式交互数字人的基础。然而,现有模型多依赖双向注意力和多步去噪,仅能生成短片段,难以满足长时间实时交互需求。本文提出 Omni-LiveAvatar,首个实现分钟级、实时流式音视频虚拟人生成的框架。具体包括:(1) 一种渐进式自回归蒸馏流程,将大型双向音视频扩散模型压缩为无辅助稳定机制的几步自回归生成器;(2) 一种同步音视频长短时记忆机制,在有限内存下保持全局一致性;(3) 一种分层滚动提示规划策略,实现语义连贯演进与平滑提示过渡。大量实验表明,Omni-LiveAvatar 可实时生成高质量、同步的分钟级虚拟人。速度上,在单张 NVIDIA H200 GPU 上较教师模型 LTX-2 提升 33×;质量上,优于加速基线,在视觉质量、音频质量、跨模态同步和人像逼真度方面均表现更优。代码已开源:https://github.com/Aoko955/Omni-LiveAvatar。

原文摘要 · Abstract (English)

Joint audio-video generative models serve as foundation for immersive and interactive digital-human generation. Nevertheless, most existing models rely on bidirectional attention and multi-step denoising and can generate only short clips, making them unsuitable for real-time interaction over extended durations. We present Omni-LiveAvatar, the first framework for minute-level, real-time streaming joint audio-video avatar generation. Specifically, we propose (1) a progressive autoregressive distillation pipeline that transfers a large bidirectional joint audio-video diffusion model into a few-step autoregressive generator without auxiliary stabilization mechanisms; (2) a synchronized audio-video long-short-term memory that preserves global consistency under a bounded memory budget; and (3) a hierarchical rolling prompt planning strategy that enables coherent semantic evolution and seamless prompt transitions. Extensive experiments show that Omni-LiveAvatar generates high-quality, synchronized minute-level avatars in real time. In terms of speed, it achieves a 33$\times$ generation speedup over its teacher, LTX-2, on a single NVIDIA H200 GPU; in terms of generation quality, it outperforms accelerated baselines across visual quality, audio quality, cross-modal synchronization, and human fidelity. Our code is available at https://github.com/Aoko955/Omni-LiveAvatar.

虚拟人生成实时生成音视频同步扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。