arXiv:2604.21668cs.CV2026-04被引 1

用规则化运动描述让大模型直接理解人体动作,效果超越现有方法。

Encoder-Free Human Motion Understanding via Structured Motion Descriptions

论文配图:Encoder-Free Human Motion Understanding via Structured Motion Descriptions
图 1 · 摘自论文原文
  • 将关节位置转为结构化自然语言描述,无需学习编码器
  • 在两个数据集上问答准确率超SOTA(BABEL-QA 66.7%,HuMMan-QA 90.1%)
  • 描述可读性强,适配多种大模型且支持可解释性分析

文本型大语言模型(LLMs)的世界知识与推理能力快速提升,但当前人体动作理解方法(如动作问答、描述生成)尚未充分挖掘其潜力。现有基于LLM的方法通常依赖专用编码器将动作特征投影到LLM嵌入空间,受限于跨模态表征与对齐。受生物力学分析启发——关节角度与身体部位运动学长期作为人体动作的精确描述语言——我们提出结构化运动描述(SMD),一种基于规则的确定性方法,将关节位置序列转换为关于关节角度、身体部位运动和整体轨迹的结构化自然语言描述。通过将动作表示为文本,SMD使LLM能直接应用其预训练中关于身体部位、空间方向和运动语义的知识进行动作推理,无需学习编码器或对齐模块。实验表明,该方法在动作问答(BABEL-QA达66.7%,HuMMan-QA达90.1%)与动作描述生成(HumanML3D上R@1为0.584,CIDEr为53.16)上均超越所有先前方法。SMD还具备实用性优势:同一文本输入可在8个不同大模型(来自6个模型族)上使用,仅需轻量级LoRA微调;其可读性表示支持对动作描述的可解释注意力分析。代码、数据及预训练LoRA适配器已公开于https://yaozhang182.github.io/motion-smd/。

原文摘要 · Abstract (English)

The world knowledge and reasoning capabilities of text-based large language models (LLMs) are advancing rapidly, yet current approaches to human motion understanding, including motion question answering and captioning, have not fully exploited these capabilities. Existing LLM-based methods typically learn motion-language alignment through dedicated encoders that project motion features into the LLM's embedding space, remaining constrained by cross-modal representation and alignment. Inspired by biomechanical analysis, where joint angles and body-part kinematics have long served as a precise descriptive language for human movement, we propose \textbf{Structured Motion Description (SMD)}, a rule-based, deterministic approach that converts joint position sequences into structured natural language descriptions of joint angles, body part movements, and global trajectory. By representing motion as text, SMD enables LLMs to apply their pretrained knowledge of body parts, spatial directions, and movement semantics directly to motion reasoning, without requiring learned encoders or alignment modules. We show that this approach goes beyond state-of-the-art results on both motion question answering (66.7\% on BABEL-QA, 90.1\% on HuMMan-QA) and motion captioning (R@1 of 0.584, CIDEr of 53.16 on HumanML3D), surpassing all prior methods. SMD additionally offers practical benefits: the same text input works across different LLMs with only lightweight LoRA adaptation (validated on 8 LLMs from 6 model families), and its human-readable representation enables interpretable attention analysis over motion descriptions. Code, data, and pretrained LoRA adapters are available at https://yaozhang182.github.io/motion-smd/.

动作理解大模型可解释性文本生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。