用连续掩码自回归生成多模态人体动作，效果超越现有方法。
OmniMotion: Multimodal Motion Generation with Continuous Masked Autoregression
- 设计连续掩码自回归运动变换器，结合门控线性注意力与RMSNorm提升稳定性。
- 在文本、语音、音乐到动作的生成任务中均优于以往方法。
- 适合需要多模态动作生成的研究者与开发者使用。
全身多模态人体动作生成面临两大挑战：构建高效的生成机制，以及整合文本、语音、音乐等多种模态。与以往采用离散掩码建模或自回归建模的方法不同，本文提出一种连续掩码自回归运动变换器，利用因果注意力捕捉人体动作的时序特性。该变换器引入门控线性注意力和RMSNorm模块，使模型聚焦关键动作，抑制异常运动及多模态分布异质性带来的不稳定性。为进一步增强生成质量与多模态泛化能力，采用DiT结构将条件信息从变换器扩散至目标动作。通过AdaLN与交叉注意力融合文本、语音和音乐信号。实验表明，本框架在文本到动作、语音到手势、音乐到舞蹈等所有模态任务上均优于现有方法。
原文摘要 · Abstract (English)
Whole-body multi-modal human motion generation poses two primary challenges: creating an effective motion generation mechanism and integrating various modalities, such as text, speech, and music, into a cohesive framework. Unlike previous methods that usually employ discrete masked modeling or autoregressive modeling, we develop a continuous masked autoregressive motion transformer, where a causal attention is performed considering the sequential nature within the human motion. Within this transformer, we introduce a gated linear attention and an RMSNorm module, which drive the transformer to pay attention to the key actions and suppress the instability caused by either the abnormal movements or the heterogeneous distributions within multi-modalities. To further enhance both the motion generation and the multimodal generalization, we employ the DiT structure to diffuse the conditions from the transformer towards the targets. To fuse different modalities, AdaLN and cross-attention are leveraged to inject the text, speech, and music signals. Experimental results demonstrate that our framework outperforms previous methods across all modalities, including text-to-motion, speech-to-gesture, and music-to-dance. The code of our method will be made public.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。