提出新型脉冲神经网络,提升视觉任务性能并降低计算开销。
Breaking Global Self-Attention Bottlenecks in Transformer-based Spiking Neural Networks with Local Structure-Aware Self-Attention

- 引入局部膨胀窗口机制,兼顾局部细节与长程依赖。
- 在Tiny-ImageNet和N-CALTECH101上分别提升4.3%和8.6%准确率。
- 适合追求能效比的脉冲神经网络在大规模视觉任务中的应用。
基于Transformer的脉冲神经网络(SNNs)结合了全局自注意力机制,表现出优异性能。然而现有方法存在两大局限:一是普遍采用最大池化层压缩特征图,仅保留最强响应,未能充分保留区域特征;二是全局自注意力涉及所有特征间的交互,导致计算冗余与二次复杂度,违背了SNN稀疏性与低功耗特性。为此,我们提出局部结构感知脉冲Transformer(LSFormer),引入脉冲响应池化(SPooling)与局部结构感知脉冲自注意力(LS-SSA)。首次通过局部膨胀窗口机制,同时捕捉局部细节与长程依赖。实验表明,相比现有先进SNN模型,LSFormer在更具挑战性的静态数据集Tiny-ImageNet与神经形态数据集N-CALTECH101上,顶1分类准确率分别提升4.3%和8.6%。结果验证了其在推动高能效脉冲模型向大规模视觉应用落地方面的潜力。
原文摘要 · Abstract (English)
Transformer-based Spiking Neural Networks (SNNs) integrate SNNs with global self-attention and have demonstrated impressive performance. However, existing Transformer-based SNNs suffer from two fundamental limitations. First, they typically employ max pooling layers to reduce the size of feature maps, but the max pooling captures only the strongest response and fails to comprehensively preserve representative regional features. Second, the global self-attention involves all global feature interactions, resulting in computational redundancy and quadratic computational complexity, thus conflicting with the sparse and energy-efficient characteristics of SNNs. To address these challenges, we develop Local Structure-Aware Spiking Transformer (LSFormer), a novel Transformer-based Spiking Neural Network that incorporates Spiking Response Pooling (SPooling) and Local Structure-Aware Spiking Self-Attention (LS-SSA). For the first time, our LSFormer leverages a local dilated window mechanism to capture both local details and long-range dependencies. Experimental results demonstrate that our LSFormer achieves state-of-the-art performance compared to existing advanced Transformer-based SNNs. Notably, on the more challenging static dataset Tiny-ImageNet and neuromorphic dataset N-CALTECH101, LSFormer substantially outperforms state-of-the-art baselines by 4.3\% and 8.6\% in top-1 classification accuracy, respectively. These results highlight the potential of LSFormer to advance energy-efficient spiking models toward practical deployment in large-scale vision applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。