提出自适应非对称梯度,提升脉冲神经网络训练稳定性和能效。
A2SG:Adaptive and Asymmetric Surrogate Gradients for Training Deep Spiking Neural Networks

- 自适应窗口动态调整时空梯度,保持方向一致性。
- 非对称梯度使膜电位高的神经元获得更大更新,降低梯度波动。
- 适用于各类SNN模型,显著提升准确率与能效,适合高效神经计算研究者。
由于代理梯度引发的陡峭损失曲面和时间不一致性,深度脉冲神经网络(SNN)的训练仍具挑战。为此,我们提出统一框架A2SG:自适应且非对称的代理梯度。自适应梯度通过调整时空适应的有效窗口,减少空间梯度差异并维持时间上的梯度方向一致性;非对称梯度根据神经元膜电位大小分配不同梯度值,使高电位神经元获得更大更新,理论证明其梯度方差低于对称代理。分析进一步揭示局部梯度变化与损失曲面曲率的直接关联,解释了为何A2SG能促进收敛至更平坦的极小值并提升泛化能力。我们在多种模型(含基于CNN与Transformer的SNN)上进行广泛实验,涵盖图像分类(静态与类脑数据集)及分割任务。结果表明,A2SG在各类任务中均持续提升准确率与能效,展现出通用且可靠的训练效果。代码已开源:https://github.com/KIST-NCL/A2SG.git。
原文摘要 · Abstract (English)
Training deep spiking neural networks (SNNs) remains challenging due to sharp loss landscapes and temporal inconsistency caused by surrogate gradients. To address these challenges, we propose a unified framework: adaptive and asymmetric surrogate gradients A2SG. The adaptive gradients adjust an effective window for spatio-temporal adaptation, reducing spatial gradient variation and maintaining directional consistency of gradients over time. The asymmetric gradients reflect neuronal dynamics by assigning larger gradients to neurons with higher membrane potentials, and we prove that they yield lower variation than symmetric surrogates. Our analysis further establishes a direct connection between local gradient variation and the curvature of the loss landscape, providing a principled explanation for how A2SG promotes convergence to flatter minima and improves generalization. We conduct extensive experiments on diverse models, including CNN-based and Transformer-based SNNs, across various tasks such as image classification using both static and neuromorphic datasets, as well as segmentation. The results demonstrate that A2SG consistently improves accuracy and energy efficiency, establishing it as a general and reliable solution for training deep SNNs. Our code is available at https://github.com/KIST-NCL/A2SG.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。