arXiv:2603.27314cs.AIcs.CV2026-03被引 7

用分块动作与音乐特征生成舞蹈,提升真实感和多样性。

TokenDance: Token-to-Token Music-to-Dance Generation with Bidirectional Mamba

  • 将舞蹈和音乐分别拆解为上下身动作与语义声学特征
  • 采用双向Mamba模型实现高效非自回归生成,速度更快
  • 适合虚拟现实与数字角色动画等实际应用

音乐到舞蹈生成在虚拟现实、舞蹈教学和数字角色动画中有广泛应用。然而,现有3D舞蹈数据集覆盖有限,导致模型只能处理少量音乐风格和编舞模式,难以泛化到真实音乐,生成舞蹈常过于简单重复,缺乏表现力和真实感。为此,我们提出TokenDance,一种两阶段音乐到舞蹈生成框架,通过双模态分词与高效的逐标记生成解决该问题。第一阶段,采用有限标量量化对舞蹈和音乐进行离散化:舞蹈动作按运动学-动力学约束分解为上身与下身成分,音乐则拆分为语义与声学特征,并使用独立代码本捕捉编舞特定结构。第二阶段,引入基于双向Mamba架构的局部-全局-局部标记到标记生成器,实现连贯动作合成、强音乐-舞蹈对齐及高效非自回归推理。大量实验表明,TokenDance在生成质量与推理速度方面均达到当前最优水平,展现出在真实音乐场景下的有效性与实用价值。

原文摘要 · Abstract (English)

Music-to-dance generation has broad applications in virtual reality, dance education, and digital character animation. However, the limited coverage of existing 3D dance datasets confines current models to a narrow subset of music styles and choreographic patterns, resulting in poor generalization to real-world music. Consequently, generated dances often become overly simplistic and repetitive, substantially degrading expressiveness and realism. To tackle this problem, we present TokenDance, a two-stage music-to-dance generation framework that explicitly addresses this limitation through dual-modality tokenization and efficient token-level generation. In the first stage, we discretize both dance and music using Finite Scalar Quantization, where dance motions are factorized into upper and lower-body components with kinematic-dynamic constraints, and music is decomposed into semantic and acoustic features with dedicated codebooks to capture choreography-specific structures. In the second stage, we introduce a Local-Global-Local token-to-token generator built on a Bidirectional Mamba backbone, enabling coherent motion synthesis, strong music-dance alignment, and efficient non-autoregressive inference. Extensive experiments demonstrate that TokenDance achieves overall state-of-the-art (SOTA) performance in both generation quality and inference speed, highlighting its effectiveness and practical value for real-world music-to-dance applications.

舞蹈生成多模态Mamba非自回归

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。