arXiv:2603.19290cs.NEcs.AI2026-03被引 1

用生物神经元机制提升脉冲变压器性能,省内存又高效。

Neural Dynamics Self-Attention for Spiking Transformers

  • 引入局部感受野动态,让脉冲注意力更聚焦邻近区域。
  • 无需存储大注意力矩阵,推理内存降低显著。
  • 在视觉任务上性能超越现有脉冲变压器,适合边缘设备。

将脉冲神经网络(SNNs)与Transformer架构结合,为边缘视觉应用提供了兼顾能效与性能的前景。然而,现有脉冲变压器面临两大挑战:(i)相比人工神经网络(ANNs)性能差距明显;(ii)推理时内存开销高。通过理论分析,我们发现二者均源于脉冲自注意力(SSA)机制:缺乏局部性偏置且需存储庞大的注意力矩阵。受生物视觉神经元局部感受野(LRF)和膜电位动态启发,我们提出LRF-Dyn方法,利用具有局部感受野的脉冲神经元计算注意力,同时降低内存需求。具体地,在SSA中引入LRF机制,增强邻近区域权重,强化局部建模能力并提升性能。在此基础上,通过电荷-放电-重置动态近似注意力计算,消除显式注意力矩阵存储,显著减少推理时内存占用。大量视觉任务实验表明,该方法在大幅降低内存开销的同时实现显著性能提升,是构建能效型脉冲变压器的关键单元。

原文摘要 · Abstract (English)

Integrating Spiking Neural Networks (SNNs) with Transformer architectures offers a promising pathway to balance energy efficiency and performance, particularly for edge vision applications. However, existing Spiking Transformers face two critical challenges: (i) a substantial performance gap compared to their Artificial Neural Networks (ANNs) counterparts and (ii) high memory overhead during inference. Through theoretical analysis, we attribute both limitations to the Spiking Self-Attention (SSA) mechanism: the lack of locality bias and the need to store large attention matrices. Inspired by the localized receptive fields (LRF) and membrane-potential dynamics of biological visual neurons, we propose LRF-Dyn, which uses spiking neurons with localized receptive fields to compute attention while reducing memory requirements. Specifically, we introduce a LRF method into SSA to assign higher weights to neighboring regions, strengthening local modeling and improving performance. Building on this, we approximate the resulting attention computation via charge-fire-reset dynamics, eliminating explicit attention-matrix storage and reducing inference-time memory. Extensive experiments on visual tasks confirm that our method reduces memory overhead while delivering significant performance improvements. These results establish it as a key unit for achieving energy-efficient Spiking Transformers.

脉冲神经网络自注意力能效计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。