提出新型脉冲自注意力机制，让神经网络更省电且准确
Spiking Transformer:Introducing Accurate Addition-Only Spiking Self-Attention for Transformer
- 用混合脉冲神经元替代传统计算，避免复杂运算
- 在ImageNet上达78.66%准确率，超越现有脉冲模型
- 适合低功耗场景的AI部署，如边缘设备
Transformer在众多任务中表现优异，归功于其自注意力机制，但能耗较高。脉冲神经网络（SNN）凭借事件驱动计算和二进制脉冲传递信息，成为节能替代方案。本文提出一种新方法——准确的仅加法脉冲自注意力（A²OS²A），将二值、ReLU和三值脉冲神经元融合，改进了传统全二值化设计。该方法无需softmax和缩放操作，保留非乘法计算优势。实验表明，基于A²OS²A的脉冲Transformer在多个数据集上优于现有SNN-Transformer，甚至在ImageNet-1K上达到78.66%准确率，显著提升性能与能效。
原文摘要 · Abstract (English)
Transformers have demonstrated outstanding performance across a wide range of tasks, owing to their self-attention mechanism, but they are highly energy-consuming. Spiking Neural Networks have emerged as a promising energy-efficient alternative to traditional Artificial Neural Networks, leveraging event-driven computation and binary spikes for information transfer. The combination of Transformers' capabilities with the energy efficiency of SNNs offers a compelling opportunity. This paper addresses the challenge of adapting the self-attention mechanism of Transformers to the spiking paradigm by introducing a novel approach: Accurate Addition-Only Spiking Self-Attention (A$^2$OS$^2$A). Unlike existing methods that rely solely on binary spiking neurons for all components of the self-attention mechanism, our approach integrates binary, ReLU, and ternary spiking neurons. This hybrid strategy significantly improves accuracy while preserving non-multiplicative computations. Moreover, our method eliminates the need for softmax and scaling operations. Extensive experiments show that the A$^2$OS$^2$A-based Spiking Transformer outperforms existing SNN-based Transformers on several datasets, even achieving an accuracy of 78.66\% on ImageNet-1K. Our work represents a significant advancement in SNN-based Transformer models, offering a more accurate and efficient solution for real-world applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。