用自蒸馏提升脉冲神经网络训练效率,降低资源消耗。
Enhanced Self-Distillation Framework for Efficient Spiking Neural Network Training
- 将中间层放电率投影到轻量ANN分支,实现知识迁移。
- 在多个数据集上实现高性能,且训练复杂度显著降低。
- 区分可靠与不可靠知识,提升训练收敛性,适合资源受限场景。
脉冲神经网络(SNN)因其稀疏激活模式,在类脑硬件上具有极高的能效优势。然而,基于代理梯度和时间反向传播(BPTT)的传统训练方法不仅性能落后于人工神经网络(ANNs),还带来线性增长的计算与内存开销。为在有限算力下实现高性能SNN训练,我们提出一种增强型自蒸馏框架,结合基于发放率的反向传播进行联合优化。具体地,将SNN中间层的放电率投影至轻量级ANN分支,利用模型自身生成的高质量知识通过ANN路径优化子结构。不同于传统自蒸馏,我们发现低质量自生成知识可能阻碍收敛,因此将教师信号解耦为可靠与不可靠成分,仅使用可靠知识指导优化。在CIFAR-10、CIFAR-100、CIFAR10-DVS和ImageNet上的大量实验表明,该方法有效降低训练复杂度,同时实现高性能SNN训练。代码已开源:https://github.com/Intelli-Chip-Lab/enhanced-self-distillation-framework-for-snn。
原文摘要 · Abstract (English)
Spiking Neural Networks (SNNs) exhibit exceptional energy efficiency on neuromorphic hardware due to their sparse activation patterns. However, conventional training methods based on surrogate gradients and Backpropagation Through Time (BPTT) not only lag behind Artificial Neural Networks (ANNs) in performance, but also incur significant computational and memory overheads that grow linearly with the temporal dimension. To enable high-performance SNN training under limited computational resources, we propose an enhanced self-distillation framework, jointly optimized with rate-based backpropagation. Specifically, the firing rates of intermediate SNN layers are projected onto lightweight ANN branches, and high-quality knowledge generated by the model itself is used to optimize substructures through the ANN pathways. Unlike traditional self-distillation paradigms, we observe that low-quality self-generated knowledge may hinder convergence. To address this, we decouple the teacher signal into reliable and unreliable components, ensuring that only reliable knowledge is used to guide the optimization of the model. Extensive experiments on CIFAR-10, CIFAR-100, CIFAR10-DVS, and ImageNet demonstrate that our method reduces training complexity while achieving high-performance SNN training. Our code is available at https://github.com/Intelli-Chip-Lab/enhanced-self-distillation-framework-for-snn.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。