用Mamba建模骨骼动作时序，提升识别精度与速度
TSkel-Mamba: Temporal Dynamic Modeling via State Space Model for Human Skeleton-based Action Recognition
- 融合Transformer与Mamba，分别处理空间与时序特征
- 引入多尺度时序交互模块，捕捉跨关节动态关联
- 在多个数据集上达顶尖性能，推理速度快
基于骨架的动作识别在计算机视觉领域受到广泛关注。受近期选择性状态空间模型（SSM）Mamba在建模一维时序序列中的成功启发，本文提出TSkel-Mamba，一种混合Transformer-Mamba框架，能够有效捕捉空间与时间动态。具体而言，该方法利用空间Transformer进行空间特征学习，同时采用Mamba进行时序建模。然而，Mamba对各通道独立使用状态空间块，难以建模通道间依赖关系。为此，我们设计了时序动态建模（TDM）模块，作为可插拔组件，集成新颖的多尺度时序交互（MTI）模块。MTI模块采用多尺度循环算子，捕捉跨通道的时序交互，这对动作识别至关重要。在NTU-RGB+D 60、NTU-RGB+D 120、NW-UCLA和UAV-Human数据集上的大量实验表明，TSkel-Mamba实现了领先性能，同时保持低推理耗时，兼具高效与高精度。
原文摘要 · Abstract (English)
Skeleton-based action recognition has garnered significant attention in the computer vision community. Inspired by the recent success of the selective state-space model (SSM) Mamba in modeling 1D temporal sequences, we propose TSkel-Mamba, a hybrid Transformer-Mamba framework that effectively captures both spatial and temporal dynamics. In particular, our approach leverages Spatial Transformer for spatial feature learning while utilizing Mamba for temporal modeling. Mamba, however, employs separate SSM blocks for individual channels, which inherently limits its ability to model inter-channel dependencies. To better adapt Mamba for skeleton data and enhance Mamba`s ability to model temporal dependencies, we introduce a Temporal Dynamic Modeling (TDM) block, which is a versatile plug-and-play component that integrates a novel Multi-scale Temporal Interaction (MTI) module. The MTI module employs multi-scale Cycle operators to capture cross-channel temporal interactions, a critical factor in action recognition. Extensive experiments on NTU-RGB+D 60, NTU-RGB+D 120, NW-UCLA and UAV-Human datasets demonstrate that TSkel-Mamba achieves state-of-the-art performance while maintaining low inference time, making it both efficient and highly effective.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。