用连续令牌统一生成与理解动作和语言,避免遗忘与抖动。
LLaMo: Scaling Pretrained Language Models for Unified Motion Understanding and Generation with Continuous Autoregressive Tokens
- 用连续潜空间编码动作,不量化,避免离散化抖动。
- 在预训练语言模型上扩展,支持零样本动作生成,实时>30帧/秒。
- 适合做动作生成、跨模态理解的科研与开发人员。
大型模型的发展推动了多模态生成与理解的统一进展。然而,动作-语言统一生成与理解的研究仍不充分。现有方法通常在有限的动作-文本配对数据上微调大语言模型(LLM),导致语言能力严重遗忘。同时,先前方法常通过量化将动作转为离散表示,引入显著抖动伪影。为此,我们提出LLaMo,一种通过特定模态的Transformer混合(MoT)架构扩展预训练语言模型的统一框架。该设计在保留基础模型语言理解能力的同时,支持可扩展的多模态适配。我们以因果连续潜在空间编码人类动作,并通过轻量级流匹配头保持解码器仅依赖的自回归预测范式,实现实时流式动作生成(>30 FPS)。结合预训练语言模型的强大语言理解能力与大规模动作-文本预训练,实验表明,LLaMo在通用场景下实现了高保真度的文本到动作生成及动作到文本描述,在零样本动作生成方面表现尤为突出,标志着迈向通用统一动作-语言大模型的重要一步。
原文摘要 · Abstract (English)
Recent progress in large models has led to significant advances in unified multimodal generation and understanding. However, the development of models that unify motion-language generation and understanding remains largely underexplored. Existing approaches often fine-tune large language models (LLMs) on paired motion-text data, which can result in catastrophic forgetting of linguistic capabilities due to the limited scale of available text-motion pairs. Furthermore, prior methods typically convert motion into discrete representations via quantization to integrate with language models, introducing substantial jitter artifacts from discrete tokenization. To address these challenges, we propose LLaMo, a unified framework that extends pretrained LLMs through a modality-specific Mixture-of-Transformers (MoT) architecture. This design inherently preserves the language understanding of the base model while enabling scalable multimodal adaptation. We encode human motion into a causal continuous latent space and maintain the next-token prediction paradigm in the decoder-only backbone through a lightweight flow-matching head, allowing for streaming motion generation in real-time (>30 FPS). Leveraging the comprehensive language understanding of pretrained LLMs and large-scale motion-text pretraining, our experiments demonstrate that LLaMo achieves high-fidelity text-to-motion generation and motion-to-text captioning in general settings, especially zero-shot motion generation, marking a significant step towards a general unified motion-language large model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。