arXiv:2410.07547cs.NEcs.AI2024-10中稿 · CVPR

提出新型脉冲神经网络在线训练模型,解决梯度错位问题

Rethinking SNN Online Training and Deployment: Gradient-Coherent Learning via Hybrid-Driven LIF Model

  • 分区域设计脉冲计算机制,分离时间梯度
  • 实验显示多指标达顶尖性能,突破传统在线训练瓶颈
  • 适合追求低功耗、高效率的SNN部署场景

脉冲神经网络(SNN)因其类脑特性与低功耗优势,被视为未来人工智能的重要方向。相比传统的时空反向传播(STBP)方法,在线训练可有效避免GPU内存爆炸风险。然而,现有在线学习框架无法解决前向与反向传播中的梯度不一致问题,导致推理性能与基于STBP的模型无差异。为此,我们提出混合驱动漏电整合-发放(HD-LIF)模型族,分别在发放阈值上下区域采用不同脉冲计算机制。理论分析表明,该学习框架能有效分离时间梯度,解决代理梯度错位问题,并实现学习精度、内存复杂度与功耗的全阶段优化。实验结果证明,本方案在多项评估指标上达到当前最优性能,突破了传统SNN在线训练与部署范式。代码已公开于https://github.com/hzc1208/HD_LIF。

原文摘要 · Abstract (English)

Spiking Neural Networks (SNNs) are considered to have enormous potential in the future development of Artificial Intelligence due to their brain-inspired and energy-efficient properties. Compared to vanilla Spatial-Temporal Back-propagation (STBP) training methods, online training can effectively avoid the risk of GPU memory explosion. However, current online learning frameworks cannot tackle the gradient discrepancy problem between the forward and backward process, merely aiming to optimize the GPU memory, resulting in no performance advantages compared to the STBP-based models in the inference stage. To address the aforementioned challenges, we propose Hybrid-Driven Leaky Integrate-and-Fire (HD-LIF) model family for efficient online learning, which respectively adopt different spiking calculation mechanism in the upper-region and lower-region of the firing threshold. We theoretically point out that our learning framework can effectively separate temporal gradients and address the misalignment problem of surrogate gradients, as well as achieving full-stage optimization towards learning precision, memory complexity and power consumption. Experimental results have demonstrated that our scheme is enable to achieve state-of-the-art performance for multiple evaluation metrics, breaking through the traditional paradigm of SNN online training and deployment. Code is available at \href{https://github.com/hzc1208/HD_LIF}{here}.

脉冲神经网络在线学习梯度对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。