用关节图像和细粒度交互提升文本到动作检索的准确与可解释性
Fine-grained Motion Retrieval via Joint-Angle Motion Images and Token-Patch Late Interaction
- 将关节信息转为伪图像，结合视觉Transformer建模局部运动特征
- 引入词级后期交互机制，使文本与动作在细粒度层面精准对齐
- 支持结果可解释，适合需要精准动作生成的应用场景
文本-动作检索旨在学习自然语言描述与3D人体运动骨架序列之间的语义对齐潜在空间，实现跨模态双向搜索。现有方法多采用双编码器框架，将运动和文本压缩为全局嵌入，丢失了细粒度局部对应关系，导致精度下降，且结果可解释性差。为此，本文提出一种可解释的基于关节角度的运动表示，将关节级局部特征映射为结构化伪图像，兼容预训练视觉变换器。针对文本到动作检索，采用词级后期交互机制MaxSim，并引入掩码语言建模正则化，增强鲁棒且可解释的文本-动作对齐。在HumanML3D和KIT-ML数据集上的大量实验表明，该方法优于当前最优文本-动作检索方法，同时提供细粒度的可解释对应关系。代码见附录。
原文摘要 · Abstract (English)
Text-motion retrieval aims to learn a semantically aligned latent space between natural language descriptions and 3D human motion skeleton sequences, enabling bidirectional search across the two modalities. Most existing methods use a dual-encoder framework that compresses motion and text into global embeddings, discarding fine-grained local correspondences, and thus reducing accuracy. Additionally, these global-embedding methods offer limited interpretability of the retrieval results. To overcome these limitations, we propose an interpretable, joint-angle-based motion representation that maps joint-level local features into a structured pseudo-image, compatible with pre-trained Vision Transformers. For text-to-motion retrieval, we employ MaxSim, a token-wise late interaction mechanism, and enhance it with Masked Language Modeling regularization to foster robust, interpretable text-motion alignment. Extensive experiments on HumanML3D and KIT-ML show that our method outperforms state-of-the-art text-motion retrieval approaches while offering interpretable fine-grained correspondences between text and motion. The code is available in the supplementary material.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。