通过语义对齐的连续隐空间，让文字生成更自然的人体动作。
MoLingo: Motion-Language Alignment for Text-to-Human Motion Generation
- 用帧级文本标签训练编码器，使语义相近的动作在隐空间中靠近。
- 采用多标记交叉注意力机制，提升动作与文本的匹配度和真实感。
- 在标准指标和用户评测中均达到新最优，适合动作生成研究者使用。
我们提出MoLingo，一种文本到人体动作（T2M）生成模型，通过在连续隐空间中去噪生成逼真、自然的人体运动。现有方法通常对整个隐向量一次性或逐帧自回归地进行扩散建模。本文探讨如何使连续运动隐空间的扩散更高效，重点解决两个问题：(1) 如何构建语义对齐的隐空间以提升扩散效果；(2) 如何最优注入文本条件以确保动作忠实于描述。为此，我们提出一种基于帧级文本标签训练的语义对齐运动编码器，使具有相似语义的隐向量在空间中彼此接近，从而增强隐空间的扩散友好性。同时，我们比较单标记条件与多标记交叉注意力方案，发现后者在动作真实性和文本-动作对齐上表现更优。结合语义对齐隐空间、自回归生成与交叉注意力文本条件，我们的模型在标准评估指标及用户研究中均达到新基准。代码与模型将公开，供后续研究与应用。
原文摘要 · Abstract (English)
We introduce MoLingo, a text-to-motion (T2M) model that generates realistic, lifelike human motion by denoising in a continuous latent space. Recent works perform latent space diffusion, either on the whole latent at once or auto-regressively over multiple latents. In this paper, we study how to make diffusion on continuous motion latents work best. We focus on two questions: (1) how to build a semantically aligned latent space so diffusion becomes more effective, and (2) how to best inject text conditioning so the motion follows the description closely. We propose a semantic-aligned motion encoder trained with frame-level text labels so that latents with similar text meaning stay close, which makes the latent space more diffusion-friendly. We also compare single-token conditioning with a multi-token cross-attention scheme and find that cross-attention gives better motion realism and text-motion alignment. With semantically aligned latents, auto-regressive generation, and cross-attention text conditioning, our model sets a new state of the art in human motion generation on standard metrics and in a user study. We will release our code and models for further research and downstream usage.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。