arXiv:2602.06251cs.CVcs.AI2026-02

提出不对称时空掩码,让骨骼动作模型学得更全面。

ASMa: Asymmetric Spatio-temporal Masking for Skeleton Action Representation Learning

  • 用两种互补掩码策略,分别覆盖高阶关节和低运动帧、低阶关节和高运动帧
  • 在三个数据集上微调提升2.7%-4.4%,迁移学习最高提升5.9%
  • 模型压缩后参数减少91.4%,边缘设备推理快3倍,适合部署

自监督学习(SSL)在基于骨骼的动作识别中表现优异,依赖数据增强来学习有意义的表示。然而,现有方法主要对高运动帧和高阶关节(如3阶或4阶关节)进行掩码,导致特征表示存在偏差且不完整,难以泛化到多样化的运动模式。为此,我们提出不对称时空掩码(ASMa),用于骨骼动作表征学习,通过组合掩码策略,充分捕捉人体动作中的全谱时空动态。ASMa采用两种互补掩码策略:一种掩码高阶关节与低运动帧,另一种掩码低阶关节与高运动帧,确保骨架表示更均衡全面。此外,引入可学习特征对齐模块,有效对齐双视图学习的表示。为适应资源受限场景,采用知识蒸馏将学习并对齐的表示压缩为轻量模型。在NTU RGB+D 60、NTU RGB+D 120和PKU-MMD数据集上的大量实验表明,该方法在微调时平均提升2.7-4.4%,迁移学习至噪声数据集最高提升5.9%,性能接近全监督基线。所提蒸馏模型实现91.4%参数压缩和3倍加速,边缘设备推理效率显著提升,具备实用部署潜力。

原文摘要 · Abstract (English)

Self-supervised learning (SSL) has shown remarkable success in skeleton-based action recognition by leveraging data augmentations to learn meaningful representations. However, existing SSL methods rely on data augmentations that predominantly focus on masking high-motion frames and high-degree joints such as joints with degree 3 or 4. This results in biased and incomplete feature representations that struggle to generalize across varied motion patterns. To address this, we propose Asymmetric Spatio-temporal Masking (ASMa) for Skeleton Action Representation Learning, a novel combination of masking to learn a full spectrum of spatio-temporal dynamics inherent in human actions. ASMa employs two complementary masking strategies: one that selectively masks high-degree joints and low-motion, and another that masks low-degree joints and high-motion frames. These masking strategies ensure a more balanced and comprehensive skeleton representation learning. Furthermore, we introduce a learnable feature alignment module to effectively align the representations learned from both masked views. To facilitate deployment in resource-constrained settings and on low-resource devices, we compress the learned and aligned representation into a lightweight model using knowledge distillation. Extensive experiments on NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD datasets demonstrate that our approach outperforms existing SSL methods with an average improvement of 2.7-4.4% in fine-tuning and up to 5.9% in transfer learning to noisy datasets and achieves competitive performance compared to fully supervised baselines. Our distilled model achieves 91.4% parameter reduction and 3x faster inference on edge devices while maintaining competitive accuracy, enabling practical deployment in resource-constrained scenarios.

骨骼动作自监督模型压缩边缘部署

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。