用新初始化方法提升脉冲神经元对复杂激活函数的逼近精度
A More Accurate Approximation of Activation Function with Few Spikes Neurons
- 基于时间依赖性设计参数初始化策略，增强脉冲神经元性能
- 在不增加放电次数前提下，显著提升对Swish等非线性函数的逼近效果
- 适合低功耗深度学习应用，尤其适用于扩散模型等高算力场景
近期的深度神经网络（如扩散模型）面临高计算需求问题，因此脉冲神经网络（SNNs）因其节能特性受到关注。然而，传统脉冲神经元（如漏积分-放电神经元）难以准确表示复杂非线性激活函数（如Swish）。为解决此问题，少放电（Few Spikes, FS）神经元被提出，但因缺乏考虑神经元特性的训练方法，逼近性能受限。本文提出基于趋势的参数初始化（Tendency-based Parameter Initialization, TBPI），利用时间依赖性初始化训练参数，有效提升FS神经元对激活函数的逼近能力。
原文摘要 · Abstract (English)
Recent deep neural networks (DNNs), such as diffusion models [1], have faced high computational demands. Thus, spiking neural networks (SNNs) have attracted lots of attention as energy-efficient neural networks. However, conventional spiking neurons, such as leaky integrate-and-fire neurons, cannot accurately represent complex non-linear activation functions, such as Swish [2]. To approximate activation functions with spiking neurons, few spikes (FS) neurons were proposed [3], but the approximation performance was limited due to the lack of training methods considering the neurons. Thus, we propose tendency-based parameter initialization (TBPI) to enhance the approximation of activation function with FS neurons, exploiting temporal dependencies initializing the training parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。