提出高效脉冲视频变压器，实现线性时间复杂度与超低功耗。
SpikeVideoFormer: An Efficient Spike-Driven Video Transformer with Hamming Attention and $\mathcal{O}(T)$ Complexity
- 设计脉冲哈明注意力机制，理论指导从传统注意力到脉冲注意力的转换
- 在动作识别等任务上性能超越现有脉冲网络15%以上，效率提升达16倍
- 适合追求低功耗视频理解的边缘设备部署，如智能摄像头、可穿戴设备
脉冲神经网络（SNN）在多种视觉任务中表现媲美人工神经网络（ANN），同时具备更高能效。然而，现有基于SNN的Transformer多聚焦于单图像任务，忽视了视频任务中SNN的效率优势。本文提出SpikeVideoFormer，一种高效的脉冲驱动视频Transformer，具有线性时间复杂度 𝒪(T)。我们设计了脉冲驱动的哈明注意力（SDHA），理论上指导从传统实值注意力向脉冲注意力的转换。基于SDHA，进一步分析多种脉冲驱动时空注意力结构，识别出最优方案，在保持线性时间复杂度的同时，显著提升视频任务性能。在分类、人体姿态追踪和语义分割等下游任务中验证了模型的泛化能力与效率。实验表明，该方法在后两项任务上相较现有SNN方法提升超15%，且性能接近最新ANN方法，同时在三项任务上分别实现×16、×10、×5的能效提升。
原文摘要 · Abstract (English)
Spiking Neural Networks (SNNs) have shown competitive performance to Artificial Neural Networks (ANNs) in various vision tasks, while offering superior energy efficiency. However, existing SNN-based Transformers primarily focus on single-image tasks, emphasizing spatial features while not effectively leveraging SNNs' efficiency in video-based vision tasks. In this paper, we introduce SpikeVideoFormer, an efficient spike-driven video Transformer, featuring linear temporal complexity $\mathcal{O}(T)$. Specifically, we design a spike-driven Hamming attention (SDHA) which provides a theoretically guided adaptation from traditional real-valued attention to spike-driven attention. Building on SDHA, we further analyze various spike-driven space-time attention designs and identify an optimal scheme that delivers appealing performance for video tasks, while maintaining only linear temporal complexity. The generalization ability and efficiency of our model are demonstrated across diverse downstream video tasks, including classification, human pose tracking, and semantic segmentation. Empirical results show our method achieves state-of-the-art (SOTA) performance compared to existing SNN approaches, with over 15\% improvement on the latter two tasks. Additionally, it matches the performance of recent ANN-based methods while offering significant efficiency gains, achieving $\times 16$, $\times 10$ and $\times 5$ improvements on the three tasks. https://github.com/JimmyZou/SpikeVideoFormer
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。