arXiv:2601.07632cs.CVcs.AI2026-01

让大模型理解动作时,用几何对齐提升推理能力。

GeoMotionGPT: Geometry-Aligned Motion Understanding with Large Language Models

  • 用正交约束统一动作码本与语言嵌入的几何结构
  • 在HumanML3D上提升平均性能22.4%,KIT-ML上14.4%
  • 适合做动作理解、动作-语言推理的研究者

离散动作分词近期使大语言模型(LLMs)成为动作理解与动作-语言推理的通用骨干。然而,现有方法通常将动作量化与语义嵌入学习解耦,仅通过令牌ID关联,未能有效对齐动作空间的内在几何结构与嵌入空间,限制了大模型进行精细动作推理的能力。本文主张,当两种模态共享统一几何基础时,对齐效果最佳。因此,我们提出一种新框架,显式在动作码本和大模型嵌入空间中强制正交性,使其关系结构自然对应。具体地,采用带有Gumbel-Softmax的解码器型量化器实现可微训练并平衡码本使用;通过稀疏投影将动作码映射至大模型嵌入空间,同时保持正交性;最后采用两阶段正交正则化调度,在量化器训练与大模型微调中施加软约束,维持几何对齐而不影响语义适应。大量实验表明,该框架在HumanML3D上比最强基线提升22.4%的综合平均性能,在KIT-ML上提升14.4%;消融实验证实了量化器、投影与正则化设计的有效性。

原文摘要 · Abstract (English)

Discrete motion tokenization has recently enabled Large Language Models (LLMs) to serve as versatile backbones for motion understanding and motion-language reasoning. However, existing pipelines typically decouple motion quantization from semantic embedding learning, linking them solely via token IDs. This approach fails to effectively align the intrinsic geometry of the motion space with the embedding space, thereby hindering the LLM's capacity for nuanced motion reasoning. We argue that alignment is most effective when both modalities share a unified geometric basis. Therefore, instead of forcing the LLM to reconstruct the complex geometry among motion tokens from scratch, we present a novel framework that explicitly enforces orthogonality on both the motion codebook and the LLM embedding space, ensuring that their relational structures naturally mirror each other. Specifically, we employ a decoder-only quantizer with Gumbel-Softmax for differentiable training and balanced codebook usage. To bridge the modalities, we use a sparse projection that maps motion codes into the LLM embedding space while preserving orthogonality. Finally, a two-stage orthonormal regularization schedule enforces soft constraints during tokenizer training and LLM fine-tuning to maintain geometric alignment without hindering semantic adaptation. Extensive experiments show that our framework improves the aggregated Average by 22.4% over the strongest baseline on HumanML3D and by 14.4% on KIT-ML, while ablations confirm the effectiveness of the tokenizer, projection, and regularization designs.

动作理解大模型几何对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。