QIF神经元比LIF神经元在脉冲梯度下降中更稳定，表现更好。
Quadratic integrate-and-fire neurons exhibit less fragmented loss landscapes and outperform leaky integrate-and-fire neurons in spike-based gradient descent
- 用QIF神经元替代LIF神经元，使脉冲梯度下降连续平滑。
- 在Spiking Heidelberg Digits数据集上，QIF模型准确率显著更高。
- 适合从事脉冲神经网络训练与类脑计算的研究者参考。
训练脉冲神经网络对于模拟生物神经网络和实现类脑计算至关重要。然而，广泛使用的漏电积分-发放（LIF）神经元对参数微小变化敏感，可能导致脉冲的突然出现或消失，破坏后续活动，造成神经表征不稳定及训练过程中永久沉默的神经元。最近研究发现，一类包含二次积分-发放（QIF）神经元的模型可避免这些不连续性，支持连续甚至光滑的脉冲梯度下降。但其实际优势尚不明确。本文通过在经典Spiking Heidelberg Digits数据集上对LIF与QIF神经元网络进行受控对比，首先通过全面超参数搜索优化两者，结果表明QIF神经元性能更优。其次，可视化损失与梯度景观发现，LIF神经元的损失景观不连续、碎片化严重，梯度震荡剧烈；分析单样本景观显示，这源于脉冲时间顺序变化引起的突变脉冲事件。整体结果支持在梯度下降训练中采用具有连续放电动态的神经元模型，如QIF神经元。
原文摘要 · Abstract (English)
The ability to train spiking neural networks is essential for modeling biological neural networks as well as for neuromorphic computing. However, for the extensively used leaky integrate-and-fire (LIF) neurons, arbitrarily small parameter changes can induce spike (dis)appearances that disrupt subsequent activity, leading to unstable neural representations and permanently silent neurons during exact spike-based gradient descent. Recent work shows that a class of neuron models, which includes the quadratic integrate-and-fire (QIF) neuron, avoids these discontinuities and enables continuous and even smooth spike-based gradient descent. However, it remains unclear whether these advantages translate into practice. Here, we demonstrate that they do so via a controlled comparison between networks of LIF and QIF neurons on the popular Spiking Heidelberg Digits dataset. Specifically, in a first step, we perform a thorough hyperparameter search to optimize both models, revealing a clear performance advantage of QIF neurons. In a second step, we visualize the loss and gradient landscapes. Consistent with their inferior performance, we find that the loss landscapes of LIF neurons, which are discontinuous, appear more fragmented and the related gradients more erratic. An analysis of the landscapes of single samples indicates that these features arise from changes in the temporal order of spikes, which often cause disruptive spike (dis)appearances. Overall, our results advocate replacing LIF neurons with neuron models exhibiting continuous spiking dynamics, such as QIF neurons, for gradient descent training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。