将人体骨骼序列转为图像，让大模型理解动作
Universal Skeleton Understanding via Differentiable Rendering and MLLMs
- 用可微渲染器把任意格式骨骼数据转成图像序列
- 在开放词汇动作识别上准确率超基线，跨格式泛化强
- 适合做动作理解、问答和描述生成的开发者
多模态大语言模型具备强大的视觉-语言推理能力，但无法直接处理人体骨骼等结构化非视觉数据。现有方法要么将骨骼动态压缩为有损特征向量，要么将运动离散化为泛化能力差的符号令牌。我们提出SkeletonLLM，通过可微、格式无关的渲染器DrAction，将任意骨骼序列转换为模型原生视觉模态的图像序列。由于整个流程端到端可微，大模型的梯度可直接指导渲染生成任务相关视觉标记。为进一步提升推理能力，引入协同训练策略：因果推理蒸馏从教师模型迁移分步推理逻辑，判别性微调强化易混淆动作间的决策边界。SkeletonLLM在开放词汇动作识别中表现优异，其学习到的推理能力自然延伸至跨格式的运动描述生成与问答任务，为大模型应用到非原生模态提供了可行路径。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) exhibit strong visual-language reasoning, yet cannot process structured, non-visual data such as human skeletons. Existing methods either compress skeleton dynamics into lossy feature vectors for text alignment, or quantize motion into discrete tokens that generalize poorly across heterogeneous skeleton formats. We present SkeletonLLM, which achieves universal skeleton understanding by translating arbitrary skeleton sequences into the MLLM's native visual modality. At its core is DrAction, a differentiable, format-agnostic renderer that converts skeletal kinematics into compact image sequences. Because the pipeline is end-to-end differentiable, MLLM gradients can directly guide the rendering to produce task-informative visual tokens. To further enhance reasoning capabilities, we introduce a cooperative training strategy: Causal Reasoning Distillation transfers structured, step-by-step reasoning from a teacher model, while Discriminative Finetuning sharpens decision boundaries between confusable actions. SkeletonLLM demonstrates strong generalization \revise{in open-vocabulary action recognition, while its learned reasoning capabilities naturally extend to motion captioning and question answering across heterogeneous skeleton formats} -- suggesting a viable path for applying MLLMs to non-native modalities. Code: https://github.com/wangzy01/SkeletonLLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。