arXiv:2603.05963cs.CVcs.AI2026-03

将骨骼序列转为图像格式,用视觉预训练模型学习骨骼表示。

Skeleton-to-Image Encoding: Enabling Skeleton Representation Learning via Vision-Pretrained Models

  • 将骨骼点按身体部位分组并重排成图像样式数据。
  • 在NTU-60/120、PKU-MMD上实现跨格式自监督学习,性能领先。
  • 适配多种骨骼数据源,无需额外分支,通用性强。

大规模预训练视觉模型在众多下游任务中表现卓越,涵盖跨模态与多模态场景。然而,由于数据格式差异,其直接应用于3D人体骨骼数据仍具挑战性。此外,缺乏大规模骨骼数据集,且在不增加额外模型分支的前提下将骨骼数据融入多模态动作识别,仍是重要研究方向。为此,我们提出骨架到图像编码(S2I),通过基于身体部位语义对关节进行分区与排列,并缩放至标准图像尺寸,将骨架序列转换为类似图像的数据。该编码首次使强大视觉预训练模型可用于自监督骨架表示学习,有效将丰富的视觉领域知识迁移至骨架分析。现有骨架方法通常针对特定同质骨架格式设计模型,忽视了来自不同数据源的结构异质性。相比之下,我们的S2I表示提供统一的图像样格式,天然兼容异构骨架数据。在NTU-60、NTU-120和PKU-MMD上的大量实验表明,该方法在自监督骨架表示学习中具有有效性与泛化能力,尤其在跨格式评估设置下表现优异。

原文摘要 · Abstract (English)

Recent advances in large-scale pretrained vision models have demonstrated impressive capabilities across a wide range of downstream tasks, including cross-modal and multi-modal scenarios. However, their direct application to 3D human skeleton data remains challenging due to fundamental differences in data format. Moreover, the scarcity of large-scale skeleton datasets and the need to incorporate skeleton data into multi-modal action recognition without introducing additional model branches present significant research opportunities. To address these challenges, we introduce Skeleton-to-Image Encoding (S2I), a novel representation that transforms skeleton sequences into image-like data by partitioning and arranging joints based on body-part semantics and resizing to standardized image dimensions. This encoding enables, for the first time, the use of powerful vision-pretrained models for self-supervised skeleton representation learning, effectively transferring rich visual-domain knowledge to skeleton analysis. While existing skeleton methods often design models tailored to specific, homogeneous skeleton formats, they overlook the structural heterogeneity that naturally arises from diverse data sources. In contrast, our S2I representation offers a unified image-like format that naturally accommodates heterogeneous skeleton data. Extensive experiments on NTU-60, NTU-120, and PKU-MMD demonstrate the effectiveness and generalizability of our method for self-supervised skeleton representation learning, including under challenging cross-format evaluation settings.

骨骼表示自监督学习视觉预训练多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。