将Transformer转为脉冲网络,实现高精度低功耗高效推理。
Towards High-performance Spiking Transformers from ANN to SNN Conversion
- 用期望补偿模块保留转换后精度,利用历史信息预测输出。
- 4个时间步达88.6%准确率,仅损失1%且功耗降至35%。
- 适合追求能效比的神经形态计算系统研发者。
脉冲神经网络(SNN)因其能效高、处理速度快和鲁棒性强而展现出巨大潜力。构建SNN主要有两种方法:直接训练需大量内存,而转换方法更简单高效。但现有转换方法多集中于卷积神经网络(CNN),将Transformer转为SNN面临非线性模块带来的挑战。本文提出期望补偿模块(ECM),通过前T个时间步的信息计算第T步的期望输出,以保持精度。同时引入多阈值神经元与并行参数归一化,解决高精度所需长时间步导致的延迟和功耗问题。实验表明,该方法在复杂数据集上达到当前最优性能:使用4个时间步时,顶-1准确率达88.60%,仅损失1%精度,功耗仅为原Transformer的35%。据我们所知,这是首个成功实现高精度、低延迟、低功耗的Spiking Transformer ANNs到SNNs转换。源码已公开于https://github.com/h-z-h-cell/Transformer-to-SNN-ECMT。
原文摘要 · Abstract (English)
Spiking neural networks (SNNs) show great potential due to their energy efficiency, fast processing capabilities, and robustness. There are two main approaches to constructing SNNs. Direct training methods require much memory, while conversion methods offer a simpler and more efficient option. However, current conversion methods mainly focus on converting convolutional neural networks (CNNs) to SNNs. Converting Transformers to SNN is challenging because of the presence of non-linear modules. In this paper, we propose an Expectation Compensation Module to preserve the accuracy of the conversion. The core idea is to use information from the previous T time-steps to calculate the expected output at time-step T. We also propose a Multi-Threshold Neuron and the corresponding Parallel Parameter normalization to address the challenge of large time steps needed for high accuracy, aiming to reduce network latency and power consumption. Our experimental results demonstrate that our approach achieves state-of-the-art performance. For example, we achieve a top-1 accuracy of 88.60\% with only a 1\% loss in accuracy using 4 time steps while consuming only 35\% of the original power of the Transformer. To our knowledge, this is the first successful Artificial Neural Network (ANN) to SNN conversion for Spiking Transformers that achieves high accuracy, low latency, and low power consumption on complex datasets. The source codes of the proposed method are available at https://github.com/h-z-h-cell/Transformer-to-SNN-ECMT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。