实现140亿参数模型的实时无限长音频驱动形象生成
Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Length
- 两阶段蒸馏+互补长时策略，解决扩散模型实时流式与长期漂移难题
- 10000秒以上稳定生成，5块H800显卡下达45帧/秒，首跳延迟1.21秒
- 首创长序列评估基准GenBench，适合音视频交互与虚拟人开发
音频驱动形象交互需要实时、流式且无限长度的生成能力，这与现有扩散模型的顺序去噪和长时漂移特性根本冲突。我们提出Live Avatar，一个算法-系统协同设计的框架，用于140亿参数扩散模型。算法层面，采用两阶段流水线将预训练双向模型蒸馏为因果、少步数的流式模型，并通过一组互补的长时策略消除身份漂移和视觉伪影，实现超过10000秒的稳定自回归生成。系统层面，引入时间步强制流水并行（TPP），使每块GPU固定负责一个去噪时间步，将顺序扩散链转化为异步空间流水线，显著提升吞吐量并改善时间一致性。Live Avatar在5块H800 GPU上实现45 FPS，首跳延迟1.21秒，据我们所知是首个实现140亿参数扩散模型实时流式无限长形象生成的方案。我们还引入GenBench，一个标准化的长时序评估基准，以促进可复现评价。项目主页见https://liveavatar.github.io/。
原文摘要 · Abstract (English)
Audio-driven avatar interaction demands real-time, streaming, and infinite-length generation -- capabilities fundamentally at odds with the sequential denoising and long-horizon drift of current diffusion models. We present Live Avatar, an algorithm-system co-designed framework that addresses both challenges for a 14-billion-parameter diffusion model. On the algorithm side, a two-stage pipeline distills a pretrained bidirectional model into a causal, few-step streaming one, while a set of complementary long-horizon strategies eliminate identity drift and visual artifacts, enabling stable autoregressive generation exceeding 10000 seconds. On the system side, Timestep-forcing Pipeline Parallelism (TPP) assigns each GPU a fixed denoising timestep, converting the sequential diffusion chain into an asynchronous spatial pipeline that simultaneously boosts throughput and improves temporal consistency. Live Avatar achieves 45 FPS with a TTFF of 1.21\,s on 5 H800 GPUs, and to our knowledge is the first to enable practical real-time streaming of a 14B diffusion model for infinite-length avatar generation. We further introduce GenBench, a standardized long-form benchmark, to facilitate reproducible evaluation. Our project page is at https://liveavatar.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。