通过自适应分配比特数,让脉冲神经网络更省电且更准确。
Towards Efficient and Accurate Spiking Neural Networks via Adaptive Bit Allocation
- 按层动态分配权重和脉冲的比特数与时间长度,实现资源精细化控制。
- 在ImageNet上比先进基线提升2.69%准确率,比特预算降低4.16倍。
- 适合追求能效与精度平衡的脉冲神经网络研究者和工程师。
多比特脉冲神经网络(SNN)近年来成为热点,旨在实现节能高效的AI。然而,随着比特数增加,内存与计算需求急剧上升,性能提升不再成比例。本文基于不同层重要性差异、额外比特可能被浪费或干扰的洞察,提出一种针对直接训练SNN的自适应比特分配策略,实现细粒度的层间资源分配。具体地,将权重和脉冲的时间长度与比特宽度参数化,并使其可通过梯度学习和控制。为应对可变比特宽度与时间长度带来的挑战,提出改进型脉冲神经元,可处理不同时间长度,支持时间长度的梯度传播,且更适配脉冲量化。此外,理论分析了可学习比特宽度的步长不匹配问题,该问题可能导致严重量化误差,并提出步长更新机制缓解此问题。在CIFAR、ImageNet等静态数据集及CIFAR-DVS、DVS-GESTURE等动态数据集上的实验表明,所提方法在降低整体内存与计算开销的同时,实现了更高精度。尤其在ImageNet上,SEWResNet-34模型相较先进基线提升2.69%准确率,比特预算降低4.16倍。代码已开源。
原文摘要 · Abstract (English)
Multi-bit spiking neural networks (SNNs) have recently become a heated research spot, pursuing energy-efficient and high-accurate AI. However, with more bits involved, the associated memory and computation demands escalate to the point where the performance improvements become disproportionate. Based on the insight that different layers demonstrate different importance and extra bits could be wasted and interfering, this paper presents an adaptive bit allocation strategy for direct-trained SNNs, achieving fine-grained layer-wise allocation of memory and computation resources. Thus, SNN's efficiency and accuracy can be improved. Specifically, we parametrize the temporal lengths and the bit widths of weights and spikes, and make them learnable and controllable through gradients. To address the challenges caused by changeable bit widths and temporal lengths, we propose the refined spiking neuron, which can handle different temporal lengths, enable the derivation of gradients for temporal lengths, and suit spike quantization better. In addition, we theoretically formulate the step-size mismatch problem of learnable bit widths, which may incur severe quantization errors to SNN, and accordingly propose the step-size renewal mechanism to alleviate this issue. Experiments on various datasets, including the static CIFAR and ImageNet datasets and the dynamic CIFAR-DVS and DVS-GESTURE datasets, demonstrate that our methods can reduce the overall memory and computation cost while achieving higher accuracy. Particularly, our SEWResNet-34 can achieve a 2.69\% accuracy gain and 4.16$\times$ lower bit budgets over the advanced baseline work on ImageNet. This work is open-sourced at \href{https://github.com/Ikarosy/Towards-Efficient-and-Accurate-Spiking-Neural-Networks-via-Adaptive-Bit-Allocation}{this link}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。