arXiv:2412.09220cs.CV2024-12AAAI被引 29

提出统一骨架表示学习框架,通过多粒度去相关提升动作识别与检测性能。

USDRL: Unified Skeleton-Based Dense Representation Learning with Multi-Grained Feature Decorrelation

  • 引入多粒度特征去相关机制,消除时间、空间和实例维度冗余。
  • 在NTU-60/120、PKU-MMD数据集上显著超越现有最先进方法。
  • 适合需要精细动作表征的密集预测任务,如动作检测与定位。

对比学习在基于骨架的表示学习中取得显著进展,但现有方法多依赖负样本,需动量编码器和内存库,增加训练难度。同时,这些方法主要关注全局表示,忽视对密集预测任务至关重要的局部细节。为此,我们提出基于特征去相关的统一骨架密集表示学习框架USDRL,通过在时间、空间和实例三个维度上进行多粒度特征去相关,减少表示维度间的冗余,最大化特征信息提取。此外,设计了密集时空编码器(DSTE),有效捕捉细粒度动作表示,显著提升密集预测任务表现。在NTU-60、NTU-120、PKU-MMD I和PKU-MMD II等多个基准上,涵盖动作识别、检索和检测等多样化下游任务的全面实验表明,本方法显著优于当前最先进水平。代码与模型已开源。

原文摘要 · Abstract (English)

Contrastive learning has achieved great success in skeleton-based representation learning recently. However, the prevailing methods are predominantly negative-based, necessitating additional momentum encoder and memory bank to get negative samples, which increases the difficulty of model training. Furthermore, these methods primarily concentrate on learning a global representation for recognition and retrieval tasks, while overlooking the rich and detailed local representations that are crucial for dense prediction tasks. To alleviate these issues, we introduce a Unified Skeleton-based Dense Representation Learning framework based on feature decorrelation, called USDRL, which employs feature decorrelation across temporal, spatial, and instance domains in a multi-grained manner to reduce redundancy among dimensions of the representations to maximize information extraction from features. Additionally, we design a Dense Spatio-Temporal Encoder (DSTE) to capture fine-grained action representations effectively, thereby enhancing the performance of dense prediction tasks. Comprehensive experiments, conducted on the benchmarks NTU-60, NTU-120, PKU-MMD I, and PKU-MMD II, across diverse downstream tasks including action recognition, action retrieval, and action detection, conclusively demonstrate that our approach significantly outperforms the current state-of-the-art (SOTA) approaches. Our code and models are available at https://github.com/wengwanjiang/USDRL.

骨架学习对比学习密集预测特征去相关

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。