arXiv:2606.20888cs.CV2026-06

用语言模型理解人体动作的精细细节,支持2D/3D输入并显式编码时间信息。

Fine-grained Human Motion Understanding with Language Models

论文配图:Fine-grained Human Motion Understanding with Language Models
图 1 · 摘自论文原文
  • 将动作表示为带时间戳的骨骼姿态序列,支持时序推理
  • 在多个基准上达到当前最佳性能,2D输入超越以往3D方法
  • 融合多种任务监督,提升对动作顺序与节奏的理解能力

本文提出一种基于大语言模型的细粒度人体动作理解方法,将运动表示为带有显式时间戳的骨骼姿态序列,每个姿态编码关节位置并具备时间锚定,使模型能够推理动作顺序、持续时间和节奏。为探究运动-语言推理所需监督类型,构建了涵盖姿态描述、姿态问答、动作描述和动作问答的多样化训练混合数据。消融实验表明,主要性能提升来自姿态与动作层面的多样化监督,分阶段训练带来较小额外收益。不同于依赖真实3D动作捕捉的以往方法,本方法通过统一的姿态编码器支持2D与3D骨骼表示,并可选融合视频提供上下文信息。在BABEL-QA、HuMMan-QA、CompMo、NTU-RGB+D和QEVD-Coach等多个数据集上的实验证明,该方法在多个基准上达到当前最优表现,凸显显式时间编码与多样化监督的有效性。值得注意的是,仅使用2D骨骼输入时,其性能仍超越此前基于3D的方法。

原文摘要 · Abstract (English)

In this work, we propose \methodname, an LLM-based model for fine-grained human motion understanding that represents motion as a sequence of skeletal poses with explicit timestamps for each pose. Each pose encodes body joint positions and is temporally grounded with timestamp tokens, allowing the model to reason about motion order, duration, and rhythm. To study what supervision is needed for motion-language reasoning, we construct a diverse training mixture spanning pose captioning, pose question answering, motion captioning, and motion question answering. Our ablations show that the primary gains come from the diversity of pose- and motion-level supervision, while staged training provides a smaller additional benefit. Different from previous works that rely on ground-truth 3D motion capture, our approach supports both 2D and 3D skeletal motion representations through a unified pose encoder, and can optionally incorporate video to provide contextual information. Extensive experiments on BABEL-QA, HuMMan-QA, CompMo, NTU-RGB+D, and QEVD-Coach demonstrate that our method achieves state-of-the-art performance across multiple benchmarks, highlighting the effectiveness of explicit temporal encoding and diverse pose- and motion-level supervision for fine-grained human motion understanding. Notably, even when using only 2D skeletal input, our approach surpasses previous 3D-based methods.

动作理解语言模型时间编码2D骨骼

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。