通过反转权重与激活位,提升脉冲神经网络精度。
ReverB-SNN: Reversing Bit of the Weight and Activation for Spiking Neural Networks
- 用实值脉冲激活搭配二值权重,保留高效计算优势。
- 在多个数据集上准确率显著优于现有方法。
- 适合追求高能效与高精度的神经网络部署场景。
脉冲神经网络(SNN)作为一种类脑神经网络架构,近年来受到广泛关注。其利用二值脉冲激活实现高效信息传输,将乘法替换为加法,从而提升能效。然而,二值脉冲激活常无法充分捕捉数据信息,导致精度下降。为此,本文提出 extbf{ReverB-SNN},受近期研究启发——量化激活比量化权重对精度影响更大,主张反转权重与激活的比特表示。具体地,该方法在SNN中采用实值脉冲激活与二值权重,既保留事件驱动和无乘法的优势,又增强激活的信息容量。同时,引入可训练因子以自适应学习权重幅值,提升网络容量。为保持推理效率,训练后的可学习二值权重通过重参数化技术还原为标准形式。大量实验在多种网络结构与静态/动态数据集上表明,本方法持续优于当前最优方法。
原文摘要 · Abstract (English)
The Spiking Neural Network (SNN), a biologically inspired neural network infrastructure, has garnered significant attention recently. SNNs utilize binary spike activations for efficient information transmission, replacing multiplications with additions, thereby enhancing energy efficiency. However, binary spike activation maps often fail to capture sufficient data information, resulting in reduced accuracy. To address this challenge, we advocate reversing the bit of the weight and activation for SNNs, called \textbf{ReverB-SNN}, inspired by recent findings that highlight greater accuracy degradation from quantizing activations compared to weights. Specifically, our method employs real-valued spike activations alongside binary weights in SNNs. This preserves the event-driven and multiplication-free advantages of standard SNNs while enhancing the information capacity of activations. Additionally, we introduce a trainable factor within binary weights to adaptively learn suitable weight amplitudes during training, thereby increasing network capacity. To maintain efficiency akin to vanilla \textbf{ReverB-SNN}, our trainable binary weight SNNs are converted back to standard form using a re-parameterization technique during inference. Extensive experiments across various network architectures and datasets, both static and dynamic, demonstrate that our approach consistently outperforms state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。