用连续隐空间建模动作,让语言与动作在双流架构中协同工作。
MotionGPT3: Human Motion as a Second Modality
- 动作转为连续隐变量,避免量化误差
- 双流结构加速训练,验证损失快4倍
- 适合需要高质量动作生成的场景
随着大语言模型的发展,统一理解和生成的多模态框架日益重要,但模态与任务增多导致复杂度上升。我们发现动作量化引入近似误差,限制动作质量;而将离散文本与连续动作统一在单一流水线中会加剧跨模态干扰。受多分支Transformer启发,我们提出MotionGPT3,一种用于动作理解与生成的双模态语言-动作模型。该模型使用变分自编码器(VAE)将原始动作编码至连续隐空间,避免量化带来的伪影,同时利用预训练语言模型的语义先验。采用共享注意力的双流Transformer,在保持模态特异性路径的同时实现可控双向信息流动,降低干扰,稳定优化,实证上加速收敛且不牺牲生成保真度。针对多模态联合训练,设计生成-对齐三阶段策略,进一步提升稳定性并减少跨任务干扰。实验表明,MotionGPT3在训练损失上实现2倍加速,验证损失最快达4倍加速,同时在标准动作理解与生成基准上保持最先进性能。
原文摘要 · Abstract (English)
With the rapid progress of large language models (LLMs), multimodal frameworks that unify understanding and generation have become promising, yet they face increasing complexity as the number of modalities and tasks grows. We observe that motion quantization introduces approximation errors that cap motion quality, and that unifying discrete text and continuous motion within a single-stream backbone amplifies cross-modal interference. Motivated by recent multi-branch Transformer designs that separate signals from different modalities, we propose MotionGPT3, a bimodal motion-language model for both understanding and generation. MotionGPT3 encodes raw motion into a continuous latent space using a variational autoencoder (VAE), thereby avoiding quantization-induced artifacts, while leveraging the semantic prior of pretrained language models. A dual-stream Transformer with shared attention preserves modality-specific routes while enabling controlled, bidirectional information flow, which reduces interference, stabilizing optimization, and empirically accelerates convergence without degrading fidelity. For multimodal joint training, a generate-then-align three-stage schedule further improves stability and limits cross-task interference. Experiments show that MotionGPT3 achieves 2x faster convergence in training loss and up to 4x faster convergence in validation, while maintaining state-of-the-art performance on standard motion understanding and motion generation benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。