用脉冲神经网络提升事件相机的人体动作识别，解决长期时序信息处理难题。
Temporal-Guided Spiking Neural Networks for Event-Based Human Action Recognition
- 分段处理与3D卷积结构增强时间信息建模能力
- 在4个数据集上超越现有脉冲网络方法，最高准确率提升8.2%
- 适合关注隐私保护、低功耗视觉识别的研究者
本文探索脉冲神经网络（SNN）与事件相机在隐私保护型人体动作识别（HAR）中的协同潜力。事件相机仅记录运动轮廓，结合SNN对时空数据的脉冲处理优势，二者具有高度兼容性。然而，先前研究受限于SNN处理长期时序信息的能力。为此，本文提出两种新框架：基于时间片段的SNN（TS-SNN）通过分段提取长期时序特征，3D卷积SNN（3D-SNN）则以3D组件替代2D空间结构，促进时序信息传递。为推动该领域研究，我们构建了新数据集FallingDetection-CeleX，使用高分辨率CeleX-V事件相机（1280×800）采集7类动作。大量实验表明，所提框架在新数据集及三个类脑数据集上均优于当前先进SNN方法，验证了其在处理长程时序信息方面的有效性。
原文摘要 · Abstract (English)
This paper explores the promising interplay between spiking neural networks (SNNs) and event-based cameras for privacy-preserving human action recognition (HAR). The unique feature of event cameras in capturing only the outlines of motion, combined with SNNs' proficiency in processing spatiotemporal data through spikes, establishes a highly synergistic compatibility for event-based HAR. Previous studies, however, have been limited by SNNs' ability to process long-term temporal information, essential for precise HAR. In this paper, we introduce two novel frameworks to address this: temporal segment-based SNN (\textit{TS-SNN}) and 3D convolutional SNN (\textit{3D-SNN}). The \textit{TS-SNN} extracts long-term temporal information by dividing actions into shorter segments, while the \textit{3D-SNN} replaces 2D spatial elements with 3D components to facilitate the transmission of temporal information. To promote further research in event-based HAR, we create a dataset, \textit{FallingDetection-CeleX}, collected using the high-resolution CeleX-V event camera $(1280 \times 800)$, comprising 7 distinct actions. Extensive experimental results show that our proposed frameworks surpass state-of-the-art SNN methods on our newly collected dataset and three other neuromorphic datasets, showcasing their effectiveness in handling long-range temporal information for event-based HAR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。