用脉冲神经网络实现低功耗端到端动作检测，首次突破性能瓶颈。
SpikeTAD: Spiking Neural Networks for End-to-End Temporal Action Detection
- 直接构建脉冲神经网络用于动作检测，跳过传统转换流程
- THUMOS14上达67.2% mAP，ActivityNet-1.3上达37.42% mAP
- 专为神经形态芯片设计，适合移动设备部署
视频理解是计算机视觉的关键任务，应用广泛。随着移动设备普及，越来越多研究致力于将视频理解模型部署于其上。然而，现有模型因体积大、功耗高难以落地。脉冲神经网络（SNNs）相比人工神经网络（ANNs）具备生物合理性与低功耗优势，尤其适用于未来移动设备中的神经形态芯片。但过长的转换时间步和严重性能下降限制了其应用。为此，本文探索将SNN应用于时序动作检测（TAD）这一重要任务，提出首个基于SNN的端到端TAD架构——SpikeTAD。在保持极低功耗的同时，SpikeTAD在THUMOS14上达到67.2%平均mAP，在ActivityNet-1.3上达到37.42% mAP，证明了低功耗TAD模型的可行性。代码已开源。
原文摘要 · Abstract (English)
Video understanding is a crucial part of computer vision, with numerous application scenarios. With the increasing popularity of mobile devices, an increasing number of efforts are trying to deploy video understanding models on them. However, existing video understanding models are difficult to deploy due to their large size and prohibitive power consumption. Spiking Neural Networks (SNNs) have shown bioplausibility and low power advantages over Artificial Neural Networks (ANNs), especially on neuromorphic chips which are regarded as essential components of future mobile devices. However, excessively long conversion time-steps and severe performance degradation problems limit their application. To solve the problems above, we explore the application of SNNs on temporal action detection (TAD), which is an important task in video understanding, and propose the first SNN-based end-to-end TAD architecture coined as SpikeTAD. While maintaining extremely low power consumption, SpikeTAD achieves an average mAP of 67.2% in THUMOS14 and 37.42% in ActivityNet-1.3, demonstrating the feasibility of a low-power TAD model. Our code is available at https://github.com/MCG-NJU/SpikeTAD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。