arXiv:2603.11746cs.CV2026-03被引 1

实现小时级实时人体动画,支持20帧每秒流畅生成。

SoulX-LiveAct: Towards Hour-Scale Real-Time Human Animation with Neighbor Forcing and ConvKV Memory

  • 用邻近帧一致性强制机制稳定训练信号。
  • 固定长度记忆结构实现无限视频生成。
  • 仅需两块H100/H200 GPU即可实时运行。

自回归扩散模型通过结合扩散建模与因果推理,为序列生成任务(如视频合成)提供了有前景的框架。尽管支持流式生成,现有方法在扩展性上仍存瓶颈。本文识别出小时级实时人体动画的两大挑战:一是多数强迫策略传播样本级表征时存在扩散状态不匹配,导致学习信号不一致且收敛不稳定;二是历史表征无界增长且缺乏结构,难以有效复用缓存状态,严重限制推理效率。为此,我们提出邻居强制(Neighbor Forcing),一种扩散步一致的自回归形式,将时间相邻帧作为同噪声条件下的潜在邻居进行传播,提供分布对齐且稳定的训练信号,同时保持自回归链中的动态漂移特性。在此基础上,引入结构化ConvKV记忆机制,将因果注意力中的键值压缩为固定长度表示,实现常数内存推理,真正支持无限长视频生成,无需依赖短期运动帧缓存。大量实验表明,该方法显著提升训练收敛性、小时级生成质量与推理效率。数值结果证明,LiveAct可实现小时级实时人体动画,在仅两块NVIDIA H100或H200 GPU上达到20 FPS实时流式推理,且在唇同步精度、人体动画质量与情感表达方面达当前最优,推理开销最低。

原文摘要 · Abstract (English)

Autoregressive (AR) diffusion models offer a promising framework for sequential generation tasks such as video synthesis by combining diffusion modeling with causal inference. Although they support streaming generation, existing AR diffusion methods struggle to scale efficiently. In this paper, we identify two key challenges in hour-scale real-time human animation. First, most forcing strategies propagate sample-level representations with mismatched diffusion states, causing inconsistent learning signals and unstable convergence. Second, historical representations grow unbounded and lack structure, preventing effective reuse of cached states and severely limiting inference efficiency. To address these challenges, we propose Neighbor Forcing, a diffusion-step-consistent AR formulation that propagates temporally adjacent frames as latent neighbors under the same noise condition. This design provides a distribution-aligned and stable learning signal while preserving drifting throughout the AR chain. Building upon this, we introduce a structured ConvKV memory mechanism that compresses the keys and values in causal attention into a fixed-length representation, enabling constant-memory inference and truly infinite video generation without relying on short-term motion-frame memory. Extensive experiments demonstrate that our approach significantly improves training convergence, hour-scale generation quality, and inference efficiency compared to existing AR diffusion methods. Numerically, LiveAct enables hour-scale real-time human animation and supports 20 FPS real-time streaming inference on as few as two NVIDIA H100 or H200 GPUs. Quantitative results demonstrate that our method attains state-of-the-art performance in lip-sync accuracy, human animation quality, and emotional expressiveness, with the lowest inference cost.

人体动画扩散模型实时生成记忆机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。