让数字人实时互动生成,支持全身动作与自然表情。
StreamAvatar: Streaming Diffusion Models for Real-Time Interactive Human Avatars
- 采用自回归蒸馏与对抗优化,实现扩散模型的实时流式输出。
- 生成质量超越现有方法,在1080p下保持30帧/秒流畅运行。
- 适合虚拟主播、游戏角色等需自然交互的实时场景。
实时流式交互数字人是数字人研究中的关键挑战。尽管基于扩散模型的人像生成已取得显著进展,但其非因果架构和高计算开销使其难以用于实时流传输。此外,现有交互方法通常仅限于头部和肩部区域,无法生成手势与身体动作。为此,我们提出一种两阶段自回归适配与加速框架,通过自回归蒸馏与对抗精炼,将高保真人体视频扩散模型适配为实时交互流。为确保长期稳定性与一致性,引入三个核心组件:参考记忆池(Reference Sink)、参考锚定位置重编码(RAPR)策略,以及一致性感知判别器。基于此框架,我们构建了一个一次训练、可交互的人体数字人模型,能够生成自然的说话与倾听行为,并伴随连贯手势。大量实验表明,该方法在生成质量、实时效率与交互自然性方面均达到当前最优水平。
原文摘要 · Abstract (English)
Real-time, streaming interactive avatars represent a critical yet challenging goal in digital human research. Although diffusion-based human avatar generation methods achieve remarkable success, their non-causal architecture and high computational costs make them unsuitable for streaming. Moreover, existing interactive approaches are typically restricted to the head-and-shoulder region, limiting their ability to produce gestures and body motions. To address these challenges, we propose a two-stage autoregressive adaptation and acceleration framework that applies autoregressive distillation and adversarial refinement to adapt a high-fidelity human video diffusion model for real-time, interactive streaming. To ensure long-term stability and consistency, we introduce three key components: a Reference Sink, a Reference-Anchored Positional Re-encoding (RAPR) strategy, and a Consistency-Aware Discriminator. Building on this framework, we develop a one-shot, interactive, human avatar model capable of generating both natural talking and listening behaviors with coherent gestures. Extensive experiments demonstrate that our method achieves state-of-the-art performance, surpassing existing approaches in generation quality, real-time efficiency, and interaction naturalness. Project page: https://streamavatar.github.io .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。