实时生成稳定长视频人像动画,3分钟保持身份一致
LiveAnimate: Stable Long-Form Streaming Human Animation in Real-Time

- 用自回归架构+三步采样,实现毫秒级实时生成
- 三分钟视频身份保留度达98.7%,质量无下降
- 适合直播、虚拟主播等需即时互动的场景
基于姿态驱动的人像动画从单张参考图像和驱动姿态流生成目标人物视频。实时生成对直播、远程呈现和虚拟化身等交互应用至关重要,但现有扩散模型系统每段视频需数分钟至数小时,无法实现响应式交互。我们提出LiveAnimate,据知是首个结合实时流式传输与千亿参数规模下稳定长序列生成的动画系统,基于140亿参数的视频扩散变换器(DiT)。采用两阶段训练:首先通过参考锚定教师强制适配,将预训练双向DiT转化为块因果自回归生成器;再通过块级自强制蒸馏将采样步数压缩至三步。为保障长时间流中外观一致性,引入姿态检索存留注意力(PR-Sink)机制,包含永久锚定首帧的静态存留、基于姿态检索的历史块动态存留及三槽滚动窗口,当姿态重复时可恢复相关外观上下文,而无需存储完整序列,使内存与每块延迟恒定。结合Ulysses序列并行与算子融合,可在两块NVIDIA H100 GPU上实现19.63 FPS流式推理。在三分钟基准测试中,LiveAnimate从30秒到最后一分钟维持近似恒定的感知质量和身份保真度,而此前系统质量显著下降或需数小时离线计算才能完成相同推演。该成果确立了交互式全身动画在质量、延迟与持续时间上的新基准。
原文摘要 · Abstract (English)
Pose-driven human animation synthesizes a video of a target person from a single reference image and a driving pose stream. Real-time generation is essential for interactive applications such as live streaming, telepresence, and virtual avatars, yet diffusion-based systems require minutes to hours per clip, precluding responsive interaction. We present LiveAnimate, to our knowledge the first animation system to combine real-time streaming with stable long-form generation at billion scale, built on a 14B-parameter video Diffusion Transformer (DiT). A two-stage training pipeline first adapts a pretrained bidirectional DiT into a block-causal autoregressive generator through Reference-Anchored Teacher-Forcing Adaptation, and then reduces the sampling budget to three steps through Block-wise Self-Forcing Distillation. To preserve appearance over extended streams, we introduce Pose-Retrieval Sink Attention (PR-Sink), a bounded KV-cache mechanism combining a Static Sink that permanently anchors the first generated block, a Dynamic Sink that holds a pose-retrieved historical block, and a three-slot Rolling Window. When a pose recurs, PR-Sink restores the relevant appearance context without retaining the entire sequence, so memory and per-block latency remain constant regardless of stream duration. Together with Ulysses sequence parallelism and operator fusion, these designs enable 19.63\,FPS streaming inference on two NVIDIA H100 GPUs. On a three-minute benchmark, LiveAnimate maintains nearly constant perceptual quality and identity from the first 30 seconds to the final minute, while prior systems degrade substantially or require hours of offline computation for the same rollout. These results establish a new operating point in quality, latency, and duration for interactive full-body animation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。