用脑启发的脉冲神经网络实现高效注意力,节能超88%。
Attention via Synaptic Plasticity is All You Need: A Biologically Inspired Spiking Neuromorphic Transformer
- 用突触可塑性模拟注意力,权重随脉冲时间动态调整。
- 在CIFAR-100上达78.08%准确率,能耗仅0.49mJ,降88.47%。
- 适合低功耗硬件部署,且能解释关注区域,透明可读。
注意力是大脑选择性关注特定信息而忽略无关内容的能力,这一生物机制启发了现代Transformer中的注意力机制。尽管当前基于Transformer的大语言模型(如GPT)性能卓越,但其训练与推理能耗巨大,碳足迹显著。传统Transformer依赖点积相似性计算注意力,而脑内注意力源于神经回路。脉冲神经网络(SNNs)为节能智能提供了一条脑启发路径。然而,现有脉冲注意力仍依赖浮点运算的点积或逐元素相似性,无法适配事件驱动的脉冲;同时保留注意力矩阵,受冯·诺依曼瓶颈限制;且与脑计算机制仍有差距。为此,本文提出脉冲STDP Transformer(S²TDPT),通过突触时序依赖可塑性(STDP)实现自注意力,将查询-键关联嵌入突触权重中。STDP是脑内存与学习的核心机制,在脉冲神经网络中天然支持存内计算,兼容非冯·诺依曼硬件。在CIFAR-10和CIFAR-100上,模型分别取得94.35%和78.08%准确率,仅需4个时间步,且在CIFAR-100上能耗为0.49mJ,相较标准ANN Transformer降低88.47%。Grad-CAM显示模型关注语义相关区域,增强可解释性。S²TDPT表明,生物启发的注意力可实现节能、硬件友好且可解释的神经形态模型。
原文摘要 · Abstract (English)
Attention is the brain's ability to selectively focus on a few specific aspects while ignoring irrelevant ones. This biological principle inspired the attention mechanism in modern Transformers. Transformers now underpin large language models (LLMs) such as GPT, but at the cost of massive training and inference energy, leading to a large carbon footprint. While brain attention emerges from neural circuits, Transformer attention relies on dot-product similarity to weight elements in the input sequence. Neuromorphic computing, especially spiking neural networks (SNNs), offers a brain-inspired path to energy-efficient intelligence. Despite recent work on attention-based spiking Transformers, the core attention layer remains non-neuromorphic. Current spiking attention (i) relies on dot-product or element-wise similarity suited to floating-point operations, not event-driven spikes; (ii) keeps attention matrices that suffer from the von Neumann bottleneck, limiting in-memory computing; and (iii) still diverges from brain-like computation. To address these issues, we propose the Spiking STDP Transformer (S$^{2}$TDPT), a neuromorphic Transformer that implements self-attention through spike-timing-dependent plasticity (STDP), embedding query--key correlations in synaptic weights. STDP, a core mechanism of memory and learning in the brain and widely studied in neuromorphic devices, naturally enables in-memory computing and supports non-von Neumann hardware. On CIFAR-10 and CIFAR-100, our model achieves 94.35\% and 78.08\% accuracy with only four timesteps and 0.49 mJ on CIFAR-100, an 88.47\% energy reduction compared to a standard ANN Transformer. Grad-CAM shows that the model attends to semantically relevant regions, enhancing interpretability. Overall, S$^{2}$TDPT illustrates how biologically inspired attention can yield energy-efficient, hardware-friendly, and explainable neuromorphic models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。