arXiv:2604.17013cs.CV2026-04

解决跨源骨架数据的动作识别难题,支持开放词汇。

Toward Universal Skeleton-Based Action Recognition across Heterogeneous Skeletons and Open Vocabularies

  • 用Transformer统一不同来源的骨架数据
  • 在多个基准上实现优异泛化性能
  • 适合实际场景中多类型骨架输入的应用

用于动作识别的骨架数据来自多种来源,包括深度传感器、基于标记的运动捕捉系统以及2D/3D姿态估计算法。这些来源生成的骨架在关节数量、拓扑结构和坐标维度上存在差异,导致骨架数据天然异质。然而,以往工作忽视了骨架数据的异质性,仅使用同质骨架构建模型。此外,开放词汇动作识别对真实应用至关重要。为此,本文研究了具有开放词汇的异质骨架动作识别这一挑战性问题。我们通过整合并精炼多个代表性大规模骨架动作数据集,构建了一个大规模异质开放词汇(HOV)骨架数据集。为实现通用骨架动作识别,提出一种基于Transformer的模型:将异质骨架标准化为统一表示,通过双流运动编码器编码多模态骨架嵌入,学习时空动作表征,并通过多粒度运动-文本对齐映射到语义空间。该对齐包含三个层次的对比学习:全局实例对齐、流特定对齐和细粒度对齐。在包含异质骨架数据的多个主流基准上的大量实验表明,所提方法兼具有效性与泛化能力。代码已开源:https://github.com/jidongkuang/Universal-Skeleton。

原文摘要 · Abstract (English)

Skeleton data used for action recognition are acquired from a wide range of sources, including depth sensors, marker-based motion capture systems, and 2D/3D pose estimators. These sources yield skeletons that differ in joint number, skeletal topology, and coordinate dimensionality, making skeleton data inherently heterogeneous. However, previous works overlook the data heterogeneity of skeletons and solely construct models using homogeneous skeletons. Moreover, open-vocabulary action recognition is also essential for real-world applications. To this end, this work studies the challenging problem of heterogeneous skeleton-based action recognition with open vocabularies. We construct a large-scale Heterogeneous Open-Vocabulary (HOV) Skeleton dataset by integrating and refining multiple representative large-scale skeleton-based action datasets. To address universal skeleton-based action recognition, we propose a Transformer-based model that standardizes heterogeneous skeletons into a unified representation, encodes multi-modal skeleton embeddings with a two-stream motion encoder to learn spatio-temporal action representations, and maps them to a semantic space through multi-grained motion-text alignment. The alignment incorporates contrastive learning at three levels: global instance alignment, stream-specific alignment, and fine-grained alignment. Extensive experiments on popular benchmarks with heterogeneous skeleton data demonstrate both the effectiveness and the generalization ability of the proposed method. Code is available at https://github.com/jidongkuang/Universal-Skeleton.

动作识别骨架数据开放词汇Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。