用外部文本提升动作描述精度,让模型学会更细粒度的动作解释。
Hierarchical Motion Captioning Utilizing External Text Data Source
- 分两步:先用大模型生成动作细节描述,再用检索机制融合外部文本
- 在三个数据集上相比顶尖方法提升6%至50%的生成质量
- 特别适合处理原始数据中没有覆盖的新动作类型
本文提出一种新方法,以增强现有动作描述生成技术。现有方法需依赖带有高层描述(如“跳绳”)的动作数据,但这类数据在现有动作-文本数据集中罕见,且缺乏低层动作描述。为此,本文采用两步式层次化方案:首先利用大语言模型为数据集中出现的每个高层描述生成详细对应描述(如“手臂伸展与双腿开合同步跳跃”),用于重新训练动作到文本的模型,使其输出包含低层细节的描述;其次引入首创的基于检索的机制,将详尽的低层描述与额外文本来源中的候选高层描述对齐,并结合动作特征生成精确的高层描述。该方法通过挖掘外部文本知识,显著提升动作描述准确性,尤其对未覆盖于现有数据集中的动作表现突出。在HumanML3D、KIT和BOTH57M三个不同数据集上的实验表明,本方法在平均性能(涵盖BLEU-1、BLEU-4、CIDEr和ROUGE-L)上较当前最优的M2T-Interpretable方法提升6%至50%。
原文摘要 · Abstract (English)
This paper introduces a novel approach to enhance existing motion captioning methods, which directly map representations of movement to high-level descriptive captions (e.g., ``a person doing jumping jacks"). The existing methods require motion data annotated with high-level descriptions (e.g., ``jumping jacks"). However, such data is rarely available in existing motion-text datasets, which additionally do not include low-level motion descriptions. To address this, we propose a two-step hierarchical approach. First, we employ large language models to create detailed descriptions corresponding to each high-level caption that appears in the motion-text datasets (e.g., ``jumping while synchronizing arm extensions with the opening and closing of legs" for ``jumping jacks"). These refined annotations are used to retrain motion-to-text models to produce captions with low-level details. Second, we introduce a pioneering retrieval-based mechanism. It aligns the detailed low-level captions with candidate high-level captions from additional text data sources, and combine them with motion features to fabricate precise high-level captions. Our methodology is distinctive in its ability to harness knowledge from external text sources to greatly increase motion captioning accuracy, especially for movements not covered in existing motion-text datasets. Experiments on three distinct motion-text datasets (HumanML3D, KIT, and BOTH57M) demonstrate that our method achieves an improvement in average performance (across BLEU-1, BLEU-4, CIDEr, and ROUGE-L) ranging from 6% to 50% compared to the state-of-the-art M2T-Interpretable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。