通过分段对齐文本与动作,实现更精准的3D人体运动生成。
SegMo: Segment-aligned Text to 3D Human Motion Generation
- 将文本和动作分解为语义片段,以实现细粒度对齐。
- 在HumanML3D数据集上达到0.553的TOP 1得分,优于基线。
- 可拓展用于动作定位与文本检索等下游任务。
从文本描述生成3D人体动作是视频游戏、虚拟现实和增强现实中的重要问题。现有方法在序列层面对齐文本与动作,忽略了模态内部的语义结构。然而,文本描述和动作序列均可自然划分为更小且语义连贯的片段,这些片段可作为原子对齐单元实现更细粒度的对应。为此,我们提出SegMo,一种新的分段对齐文本条件人体动作生成框架。该框架包含三个模块:(1) 文本分段提取,将复杂文本描述分解为按时间排序的短语,每段代表一个简单原子动作;(2) 动作分段提取,将完整动作序列划分为对应的动作片段;(3) 细粒度文本-动作对齐,使用对比学习对齐文本与动作片段。大量实验表明,SegMo在两个常用数据集上优于强基线,在HumanML3D测试集上取得0.553的TOP 1分数。此外,得益于文本与动作片段共享的嵌入空间,SegMo还可应用于动作定位和动作到文本检索等检索类任务。
原文摘要 · Abstract (English)
Generating 3D human motions from textual descriptions is an important research problem with broad applications in video games, virtual reality, and augmented reality. Recent methods align the textual description with human motion at the sequence level, neglecting the internal semantic structure of modalities. However, both motion descriptions and motion sequences can be naturally decomposed into smaller and semantically coherent segments, which can serve as atomic alignment units to achieve finer-grained correspondence. Motivated by this, we propose SegMo, a novel Segment-aligned text-conditioned human Motion generation framework to achieve fine-grained text-motion alignment. Our framework consists of three modules: (1) Text Segment Extraction, which decomposes complex textual descriptions into temporally ordered phrases, each representing a simple atomic action; (2) Motion Segment Extraction, which partitions complete motion sequences into corresponding motion segments; and (3) Fine-grained Text-Motion Alignment, which aligns text and motion segments with contrastive learning. Extensive experiments demonstrate that SegMo improves the strong baseline on two widely used datasets, achieving an improved TOP 1 score of 0.553 on the HumanML3D test set. Moreover, thanks to the learned shared embedding space for text and motion segments, SegMo can also be applied to retrieval-style tasks such as motion grounding and motion-to-text retrieval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。