arXiv:2410.16746cs.CVcs.AI2024-10被引 13

用脉冲网络与Mamba结合,高效识别事件相机的动作。

SpikMamba: When SNN meets Mamba in Event-based Human Action Recognition

  • 脉冲神经网络+Mamba建模稀疏高时序事件数据
  • 在4个数据集上超越当前最优结果1.45%~7.22%
  • 适合低功耗、高动态范围场景的动作识别应用

人类动作识别(HAR)在视频分析、监控、自动驾驶、机器人和医疗等领域至关重要。主流算法基于RGB图像,但存在隐私问题。事件相机通过稀疏捕捉像素级亮度变化,不记录完整图像,具备高动态范围,能有效应对低光或高对比度等复杂光照条件。然而,事件数据空间稀疏且时间分辨率高,建模难度大。为此,我们提出SpikMamba框架,融合脉冲神经网络的能效优势与Mamba的长序列建模能力,高效捕获全局特征。同时引入基于脉冲窗口的线性注意力机制,提升局部建模精度。大量实验表明,SpikMamba在PAF、HARDVS、DVS128和E-FAction数据集上分别领先于现有最佳方法1.45%、7.22%、0.15%和3.92%。代码已开源。

原文摘要 · Abstract (English)

Human action recognition (HAR) plays a key role in various applications such as video analysis, surveillance, autonomous driving, robotics, and healthcare. Most HAR algorithms are developed from RGB images, which capture detailed visual information. However, these algorithms raise concerns in privacy-sensitive environments due to the recording of identifiable features. Event cameras offer a promising solution by capturing scene brightness changes sparsely at the pixel level, without capturing full images. Moreover, event cameras have high dynamic ranges that can effectively handle scenarios with complex lighting conditions, such as low light or high contrast environments. However, using event cameras introduces challenges in modeling the spatially sparse and high temporal resolution event data for HAR. To address these issues, we propose the SpikMamba framework, which combines the energy efficiency of spiking neural networks and the long sequence modeling capability of Mamba to efficiently capture global features from spatially sparse and high a temporal resolution event data. Additionally, to improve the locality of modeling, a spiking window-based linear attention mechanism is used. Extensive experiments show that SpikMamba achieves remarkable recognition performance, surpassing the previous state-of-the-art by 1.45%, 7.22%, 0.15%, and 3.92% on the PAF, HARDVS, DVS128, and E-FAction datasets, respectively. The code is available at https://github.com/Typistchen/SpikMamba.

事件相机动作识别脉冲网络Mamba

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。