提出小于1比特的脉冲神经网络,显著降低存储与计算开销。
S$^2$NN: Sub-bit Spiking Neural Networks
- 用不足1比特表示权重,通过聚类和异常值感知量化优化编码。
- 在图像任务上超越现有量化SNN,在极低资源下保持高精度。
- 适合边缘设备部署,尤其对能效要求高的场景有重要意义。
脉冲神经网络(SNNs)为机器智能提供了一种节能范式,但其大规模部署仍面临资源限制。尽管二值化SNN已有进展,大模型仍需大量存储与计算。为此,本文提出子比特脉冲神经网络(S²NN),以低于1比特的精度表示权重。首先基于训练良好的二值化SNN中卷积核的聚类模式建立基准,该基准虽高效却存在异常值引发的码字选择偏差。为此,提出异常值感知子比特权重量化(OS-Quant),通过识别并自适应缩放异常值优化码字选择。此外,提出基于膜电位的特征蒸馏(MPFD),通过教师模型更精准地指导高度压缩的S²NN。大量视觉任务实验表明,S²NN在性能与效率上均优于现有量化SNN,极具边缘计算应用潜力。
原文摘要 · Abstract (English)
Spiking Neural Networks (SNNs) offer an energy-efficient paradigm for machine intelligence, but their continued scaling poses challenges for resource-limited deployment. Despite recent advances in binary SNNs, the storage and computational demands remain substantial for large-scale networks. To further explore the compression and acceleration potential of SNNs, we propose Sub-bit Spiking Neural Networks (S$^2$NNs) that represent weights with less than one bit. Specifically, we first establish an S$^2$NN baseline by leveraging the clustering patterns of kernels in well-trained binary SNNs. This baseline is highly efficient but suffers from \textit{outlier-induced codeword selection bias} during training. To mitigate this issue, we propose an \textit{outlier-aware sub-bit weight quantization} (OS-Quant) method, which optimizes codeword selection by identifying and adaptively scaling outliers. Furthermore, we propose a \textit{membrane potential-based feature distillation} (MPFD) method, improving the performance of highly compressed S$^2$NN via more precise guidance from a teacher model. Extensive results on vision tasks reveal that S$^2$NN outperforms existing quantized SNNs in both performance and efficiency, making it promising for edge computing applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。