用关键帧+残差帧融合，让脉冲网络高效识别动作
ReSpike: Residual Frames-based Hybrid Spiking Neural Networks for Efficient Action Recognition
- 将视频拆成关键帧和残差帧，分别用ANN与SNN处理
- 在HMDB-51等数据集上比顶尖脉冲网络高30%以上准确率
- 适合低功耗视频动作识别场景，兼顾精度与能效
脉冲神经网络（SNN）在静态图像任务中展现出优异的能效优势，但在处理密集帧序列的视频分类任务中仍显著落后于传统人工神经网络（ANN）。为弥合这一差距，本文提出ReSpike，一种结合ANN与SNN优势的混合框架，用于高效动作识别。该方法将视频片段分解为空间成分（RGB关键帧）与时间成分（类事件残差帧），由ANN学习空间信息，由SNN学习时间动态。此外，设计多尺度交叉注意力机制实现有效特征融合。相比现有SNN基线，ReSpike在HMDB-51、UCF-101和Kinetics-400上实现超过30%的绝对准确率提升；同时在性能上接近先进ANN方法，但具备更优的准确率-能耗权衡。
原文摘要 · Abstract (English)
Spiking Neural Networks (SNNs) have emerged as a compelling, energy-efficient alternative to traditional Artificial Neural Networks (ANNs) for static image tasks such as image classification and segmentation. However, in the more complex video classification domain, SNN-based methods fall considerably short of ANN-based benchmarks due to the challenges in processing dense frame sequences. To bridge this gap, we propose ReSpike, a hybrid framework that synergizes the strengths of ANNs and SNNs to tackle action recognition tasks with high accuracy and low energy cost. By decomposing film clips into spatial and temporal components, i.e., RGB image Key Frames and event-like Residual Frames, ReSpike leverages ANN for learning spatial information and SNN for learning temporal information. In addition, we propose a multi-scale cross-attention mechanism for effective feature fusion. Compared to state-of-the-art SNN baselines, our ReSpike hybrid architecture demonstrates significant performance improvements (e.g., >30% absolute accuracy improvement on HMDB-51, UCF-101, and Kinetics-400). Furthermore, ReSpike achieves comparable performance with prior ANN approaches while bringing better accuracy-energy tradeoff.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。