利用脉冲网络的时序特性实现无需外部模型的自蒸馏,提升性能与鲁棒性。
Synergy Between the Strong and the Weak: Spiking Neural Networks are Inherently Self-Distillers
- 将SNN每时刻视为子模型,通过置信度识别强弱关系,实现自蒸馏。
- 强→弱与弱→强双向蒸馏均有效提升分类准确率,最高增益达4.2%。
- 方法无需额外训练开销,适合低功耗神经网络高效训练场景。
类脑脉冲神经网络(SNN)有望成为计算密集型人工神经网络(ANN)的低功耗替代方案,但性能差距仍存。近期研究通过知识蒸馏提升SNN性能,但依赖大教师模型或增加训练开销。本文表明,SNN可自然分解为多个子模型以实现高效自蒸馏:将每个时间步实例视为子模型,并评估其输出置信度,从而高效识别强弱关系。基于此,提出两种自蒸馏方案:(1) 强→弱:训练中强模型指导弱模型,显著提升整体性能;(2) 弱→强:弱模型反向蒸馏强模型,提取深层暗知识,同样带来显著性能提升。两种方案支持集成、并行与级联等灵活实现。实验表明,该方法有效增强SNN判别力与整体性能,同时提升对抗鲁棒性,得益于自蒸馏带来的稳定性。该工作巧妙利用SNN的时序特性,为高效训练高性能SNN提供了新思路。
原文摘要 · Abstract (English)
Brain-inspired spiking neural networks (SNNs) promise to be a low-power alternative to computationally intensive artificial neural networks (ANNs), although performance gaps persist. Recent studies have improved the performance of SNNs through knowledge distillation, but rely on large teacher models or introduce additional training overhead. In this paper, we show that SNNs can be naturally deconstructed into multiple submodels for efficient self-distillation. We treat each timestep instance of the SNN as a submodel and evaluate its output confidence, thus efficiently identifying the strong and the weak. Based on this strong and weak relationship, we propose two efficient self-distillation schemes: (1) \textbf{Strong2Weak}: During training, the stronger "teacher" guides the weaker "student", effectively improving overall performance. (2) \textbf{Weak2Strong}: The weak serve as the "teacher", distilling the strong in reverse with underlying dark knowledge, again yielding significant performance gains. For both distillation schemes, we offer flexible implementations such as ensemble, simultaneous, and cascade distillation. Experiments show that our method effectively improves the discriminability and overall performance of the SNN, while its adversarial robustness is also enhanced, benefiting from the stability brought by self-distillation. This ingeniously exploits the temporal properties of SNNs and provides insight into how to efficiently train high-performance SNNs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。