用原子动作中间表示,实现文本到任意动作的开放词汇生成
Textual Decomposition Then Sub-motion-space Scattering for Open-Vocabulary Motion Generation
- 将文本分解为原子动作,再通过组合学习构建完整动作空间
- 在多个数据集上显著超越现有方法,开放域泛化能力提升明显
- 适合需要灵活生成新动作的动画、游戏与机器人领域
文本到动作生成是计算机视觉中的关键任务,旨在根据给定文本生成目标3D动作。现有标注数据集规模有限,导致多数方法过拟合小数据集,难以泛化至开放域动作。部分方法尝试通过对齐CLIP空间或采用预训练-微调范式解决开放词汇问题,但受限于数据规模,仅能实现子文本空间到子动作空间的映射,而非全文本空间到全动作空间(全映射)的建模,而全映射是实现开放词汇生成的关键。为此,本文提出以原子动作(短时长的单体部位运动)作为中间表示,通过有序耦合的两步法——文本分解与子动作空间散射——解决全映射问题。文本分解设计细粒度描述转换算法,结合大语言模型的泛化能力,将任意运动文本转化为原子文本;子动作空间散射则学习从原子动作到目标动作的组合过程,使学习得到的子动作空间分布扩展为完整动作空间,将外推问题转化为内插问题,显著提升泛化能力。所提出的DSO-Net网络融合文本分解与子动作空间散射,实验证明其在开放词汇动作生成任务中显著优于当前最优方法。代码已公开于https://vankouf.github.io/DSONet/。
原文摘要 · Abstract (English)
Text-to-motion generation is a crucial task in computer vision, which generates the target 3D motion by the given text. The existing annotated datasets are limited in scale, resulting in most existing methods overfitting to the small datasets and unable to generalize to the motions of the open domain. Some methods attempt to solve the open-vocabulary motion generation problem by aligning to the CLIP space or using the Pretrain-then-Finetuning paradigm. However, the current annotated dataset's limited scale only allows them to achieve mapping from sub-text-space to sub-motion-space, instead of mapping between full-text-space and full-motion-space (full mapping), which is the key to attaining open-vocabulary motion generation. To this end, this paper proposes to leverage the atomic motion (simple body part motions over a short time period) as an intermediate representation, and leverage two orderly coupled steps, i.e., Textual Decomposition and Sub-motion-space Scattering, to address the full mapping problem. For Textual Decomposition, we design a fine-grained description conversion algorithm, and combine it with the generalization ability of a large language model to convert any given motion text into atomic texts. Sub-motion-space Scattering learns the compositional process from atomic motions to the target motions, to make the learned sub-motion-space scattered to form the full-motion-space. For a given motion of the open domain, it transforms the extrapolation into interpolation and thereby significantly improves generalization. Our network, $DSO$-Net, combines textual $d$ecomposition and sub-motion-space $s$cattering to solve the $o$pen-vocabulary motion generation. Extensive experiments demonstrate that our DSO-Net achieves significant improvements over the state-of-the-art methods on open-vocabulary motion generation. Code is available at https://vankouf.github.io/DSONet/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。