arXiv:2410.00580cs.CV2024-10被引 3

提出专用于脉冲神经网络的权重初始化方法,解决深层网络信息丢失问题。

Deep activity propagation via weight initialization in spiking neural networks

  • 基于脉冲网络量化特性设计新初始化策略,确保活动信号深层传播
  • 100层网络实验中实现无脉冲消失,MNIST上准确率更高、收敛更快
  • 对网络宽度和神经元参数变化具有强鲁棒性,适合实际部署

脉冲神经网络(SNNs)和类脑计算具有稀疏性和超低功耗等生物启发优势,是传统神经网络的有前途替代方案。然而,从零训练深层SNN仍具挑战,因其将连续膜电位量化为二进制脉冲,易导致信息损失与深层脉冲消失。尽管权重初始化对深度网络训练至关重要,但适用于深层SNN的有效初始状态尚不明确。现有针对传统网络(ANNs)的初始化方法常被直接用于SNN,未考虑其独特计算特性。本文推导出一种专为SNN设计的最优权重初始化方法,考虑了量化操作的影响。理论证明,该方法可避免标准方法下的脉冲衰减,实现深层活动信号无损传播。数值模拟显示,在多达100层、多时间步长的SNN中均表现良好。深入分析了层宽与神经元超参数对理论结果应用的影响。在MNIST上的实验表明,采用该初始化方案后,准确率更高且收敛更快。此外,新方法对多种网络与神经元超参数变化具有鲁棒性。

原文摘要 · Abstract (English)

Spiking Neural Networks (SNNs) and neuromorphic computing offer bio-inspired advantages such as sparsity and ultra-low power consumption, providing a promising alternative to conventional networks. However, training deep SNNs from scratch remains a challenge, as SNNs process and transmit information by quantizing the real-valued membrane potentials into binary spikes. This can lead to information loss and vanishing spikes in deeper layers, impeding effective training. While weight initialization is known to be critical for training deep neural networks, what constitutes an effective initial state for a deep SNN is not well-understood. Existing weight initialization methods designed for conventional networks (ANNs) are often applied to SNNs without accounting for their distinct computational properties. In this work we derive an optimal weight initialization method specifically tailored for SNNs, taking into account the quantization operation. We show theoretically that, unlike standard approaches, this method enables the propagation of activity in deep SNNs without loss of spikes. We demonstrate this behavior in numerical simulations of SNNs with up to 100 layers across multiple time steps. We present an in-depth analysis of the numerical conditions, regarding layer width and neuron hyperparameters, which are necessary to accurately apply our theoretical findings. Furthermore, our experiments on MNIST demonstrate higher accuracy and faster convergence when using the proposed weight initialization scheme. Finally, we show that the newly introduced weight initialization is robust against variations in several network and neuron hyperparameters.

脉冲神经网络权重初始化类脑计算深层网络

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。