用ADMM方法训练脉冲神经网络，解决传统反向传播的精度与可扩展性问题。
ADMM-Based Training for Spiking Neural Networks
- 将脉冲神经网络训练转化为ADMM迭代优化问题，避开梯度近似
- 推导出闭式更新公式，实证证明算法收敛且具潜力
- 为复杂结构和深层网络训练提供新思路，适合研究神经形态计算者
近年来，脉冲神经网络（SNNs）因其在时序处理中的高潜力与极低能耗而备受关注。然而，其仍缺乏专用且高效的训练算法。目前主流的基于替代梯度的反向传播方法（源自随机梯度下降）在应用于SNN时存在诸多缺陷：替代梯度引入的近似导致数值不精确、训练时难以准确追踪脉冲发放时间，进而影响可扩展性。本文提出一种基于交替方向乘子法（ADMM）的新SNN训练方法。该方法通过完全不同于梯度反向传播的全新思路，解决SNN阶跃函数不可导的问题。首次将SNN训练问题建模为基于ADMM的迭代优化，并推导出闭式更新公式，实证验证了优化器的收敛性、巨大潜力，并探讨了未来在不同层类型和更深架构中改进该方法的前景。
原文摘要 · Abstract (English)
In recent years, spiking neural networks (SNNs) have gained momentum due to their high potential in time-series processing combined with minimal energy consumption. However, they still lack a dedicated and efficient training algorithm. The popular backpropagation with surrogate gradients, adapted from stochastic gradient descent (SGD)-derived algorithms, has several drawbacks when used as an optimizer for SNNs. Specifically, the approximation introduced by the use of surrogate gradients leads to numerical imprecision, poor tracking of SNN firing times at training time, and, in turn, poor scalability. In this paper, we propose a novel SNN training method based on the alternating direction method of multipliers (ADMM). Our ADMM-based training aims to solve the problem of the SNN step function's non-differentiability by taking an entirely new approach with respect to gradient backpropagation. For the first time, we formulate the SNN training problem as an ADMM-based iterative optimization, derive closed-form updates, and empirically show the optimizer's convergence, its great potential, and discuss future and promising research directions to improve the method to different layer types and deeper architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。