通过时空联合密度识别关键骨骼点,提升动作识别效果
Spatio-Temporal Joint Density Driven Learning for Skeleton-Based Action Recognition
- 提出时空联合密度新指标,捕捉动态与静态骨骼的交互
- 在NTU RGB+D 120上分别提升3.5和3.6个百分点
- 适合做骨骼动作识别的自监督学习研究者参考
基于骨架的动作分类中,传统无监督或自监督方法主要关注骨架序列的动态特性。然而,骨架中运动与静止部分之间的复杂交互蕴含着尚未被充分利用的判别潜力。本文提出一种新度量——时空联合密度(STJD),用于量化这种交互。通过追踪动作过程中该密度的变化,可识别出具有判别性的关键运动/静止关节(称为“主关节”),并引导自监督学习。提出一种名为STJD-CL的对比学习策略,将骨架序列表示与其主关节表示对齐,同时对比主关节与非主关节的表示。此外,结合重建框架提出STJD-MP方法以实现更高效的学习。在NTU RGB+D 60、NTU RGB+D 120和PKUMMD数据集上的实验表明,所提方法性能显著提升,尤其在NTU RGB+D 120数据集上,使用X-sub和X-set评估时,分别较现有最优对比方法提升3.5和3.6个百分点。
原文摘要 · Abstract (English)
Traditional approaches in unsupervised or self supervised learning for skeleton-based action classification have concentrated predominantly on the dynamic aspects of skeletal sequences. Yet, the intricate interaction between the moving and static elements of the skeleton presents a rarely tapped discriminative potential for action classification. This paper introduces a novel measurement, referred to as spatial-temporal joint density (STJD), to quantify such interaction. Tracking the evolution of this density throughout an action can effectively identify a subset of discriminative moving and/or static joints termed "prime joints" to steer self-supervised learning. A new contrastive learning strategy named STJD-CL is proposed to align the representation of a skeleton sequence with that of its prime joints while simultaneously contrasting the representations of prime and nonprime joints. In addition, a method called STJD-MP is developed by integrating it with a reconstruction-based framework for more effective learning. Experimental evaluations on the NTU RGB+D 60, NTU RGB+D 120, and PKUMMD datasets in various downstream tasks demonstrate that the proposed STJD-CL and STJD-MP improved performance, particularly by 3.5 and 3.6 percentage points over the state-of-the-art contrastive methods on the NTU RGB+D 120 dataset using X-sub and X-set evaluations, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。