用Mamba-2模型实现高效语音驱动手势生成,速度提升2-4倍。
DiM-Gestor: Co-Speech Gesture Generation with Adaptive Layer Normalization Mamba-2
- 采用Mamba-2与自适应层归一化,实现低复杂度端到端建模
- 在中文手势数据集上推理速度提升2至4倍,内存减少约2.4倍
- 适合虚拟人、数字主播等需高效手势生成的应用场景
基于Transformer的语音驱动手势生成在虚拟人领域快速发展,但其二次时间与空间复杂度限制了可扩展性。为此,本文提出DiM-Gestor,一种基于Mamba-2架构的端到端生成模型。该模型包含双组件:(1)融合中文预训练模型的模糊特征提取器,自动捕捉连续语音隐含特征;(2)基于AdaLN增强的Mamba-2的语音到手势映射模块,统一处理序列中所有标记的变换,精准建模语音与手势动态的细微关联。采用扩散模型进行训练与推断,生成多样化手势输出。在新发布的中文共言语手势数据集CCG上进行的主客观评估表明,相比Transformer架构,本方法在保持竞争力结果的同时,内存使用降低约2.4倍,推理速度提升2至4倍。此外,本文发布了包含15.97小时三维全身骨骼动作的CCG数据集,涵盖五种场景下六种风格,由专业中文电视播音员录制。
原文摘要 · Abstract (English)
Speech-driven gesture generation using transformer-based generative models represents a rapidly advancing area within virtual human creation. However, existing models face significant challenges due to their quadratic time and space complexities, limiting scalability and efficiency. To address these limitations, we introduce DiM-Gestor, an innovative end-to-end generative model leveraging the Mamba-2 architecture. DiM-Gestor features a dual-component framework: (1) a fuzzy feature extractor and (2) a speech-to-gesture mapping module, both built on the Mamba-2. The fuzzy feature extractor, integrated with a Chinese Pre-trained Model and Mamba-2, autonomously extracts implicit, continuous speech features. These features are synthesized into a unified latent representation and then processed by the speech-to-gesture mapping module. This module employs an Adaptive Layer Normalization (AdaLN)-enhanced Mamba-2 mechanism to uniformly apply transformations across all sequence tokens. This enables precise modeling of the nuanced interplay between speech features and gesture dynamics. We utilize a diffusion model to train and infer diverse gesture outputs. Extensive subjective and objective evaluations conducted on the newly released Chinese Co-Speech Gestures dataset corroborate the efficacy of our proposed model. Compared with Transformer-based architecture, the assessments reveal that our approach delivers competitive results and significantly reduces memory usage, approximately 2.4 times, and enhances inference speeds by 2 to 4 times. Additionally, we released the CCG dataset, a Chinese Co-Speech Gestures dataset, comprising 15.97 hours (six styles across five scenarios) of 3D full-body skeleton gesture motion performed by professional Chinese TV broadcasters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。