首个可实时生成全身伴语动作的流式模型,支持任意长度输入。
LiveGesture Streamable Co-Speech Gesture Generation Model
- 分区域建模+因果自回归,实现流式动作生成
- 零前瞻条件下实时生成同步、多样的全身动作
- 适合虚拟主播、交互式动画等实时场景
我们提出LiveGesture,首个完全可流式处理、语音驱动的全身动作生成框架,支持零前瞻且可处理任意长度序列。不同于现有方法(多为离线生成,或独立处理身体部位或全局纠缠关节),LiveGesture从零开始设计为因果性、区域协同的动作生成系统。其包含两个核心模块:流式向量量化动作分词器(SVQ)与分层自回归变压器(HAR)。SVQ将各身体区域的动作序列转为因果离散动作标记,支持实时流式解码;HAR采用区域专家自回归变换器,建模每个区域的精细动态;再通过因果时空融合模块(xAR Fusion)整合跨区域相关运动。xAR和xAR Fusion均由持续输入的音频信号驱动,经流式因果音频编码器编码。为提升流式噪声与预测误差下的鲁棒性,引入自回归掩码训练,结合不确定性引导标记掩码与随机区域掩码,使模型在训练中暴露于部分错误的历史。在BEAT2数据集上的实验表明,LiveGesture在真实零前瞻条件下能实时生成连贯、多样且与节拍同步的全身动作,性能达到或超越当前最优离线方法。
原文摘要 · Abstract (English)
We propose LiveGesture, the first fully streamable, speech-driven full-body gesture generation framework that operates with zero look-ahead and supports arbitrary sequence length. Unlike existing co-speech gesture methods, which are designed for offline generation and either treat body regions independently or entangle all joints within a single model, LiveGesture is built from the ground up for causal, region-coordinated motion generation. LiveGesture consists of two main modules: the Streamable Vector Quantized Motion Tokenizer (SVQ) and the Hierarchical Autoregressive Transformer (HAR). The SVQ tokenizer converts the motion sequence of each body region into causal, discrete motion tokens, enabling real-time, streamable token decoding. On top of SVQ, HAR employs region-expert autoregressive (xAR) transformers to model expressive, fine-grained motion dynamics for each body region. A causal spatio-temporal fusion module (xAR Fusion) then captures and integrates correlated motion dynamics across regions. Both xAR and xAR Fusion are conditioned on live, continuously arriving audio signals encoded by a streamable causal audio encoder. To enhance robustness under streaming noise and prediction errors, we introduce autoregressive masking training, which leverages uncertainty-guided token masking and random region masking to expose the model to imperfect, partially erroneous histories during training. Experiments on the BEAT2 dataset demonstrate that LiveGesture produces coherent, diverse, and beat-synchronous full-body gestures in real time, matching or surpassing state-of-the-art offline methods under true zero look-ahead conditions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。