针对脉冲神经网络训练中知识蒸馏的匹配难题,提出两种新策略提升性能。
A Closer Look at Knowledge Distillation in Spiking Neural Network Training
- 用显著性激活图对齐师生模型特征,增强语义一致性
- 通过高斯噪声平滑学生模型稀疏输出,更好匹配教师连续预测
- 在多个数据集上验证有效,适合做SNN高效训练的研究者
脉冲神经网络(SNN)因能效优异而受到关注,但其有效训练仍具挑战。近期研究引入知识蒸馏(KD),以预训练人工神经网络(ANN)为教师,目标SNN为学生。通常采用逐元素对齐中间特征和输出概率,但忽略了两者架构本质差异:ANN输出为连续分布,而SNN输出具有稀疏性和离散性。为此,本文提出两种新策略:首先,提出显著性缩放激活图蒸馏(SAMD),将学生SNN的脉冲激活图与教师ANN的类别感知激活图对齐,引导学生学习更具语义一致性的显著性特征;其次,提出噪声平滑逻辑蒸馏(NLD),利用高斯噪声平滑学生SNN的稀疏逻辑值,促进与教师连续逻辑值的对齐。大量实验在多个数据集上验证了方法的有效性。代码已开源。
原文摘要 · Abstract (English)
Spiking Neural Networks (SNNs) become popular due to excellent energy efficiency, yet facing challenges for effective model training. Recent works improve this by introducing knowledge distillation (KD) techniques, with the pre-trained artificial neural networks (ANNs) used as teachers and the target SNNs as students. This is commonly accomplished through a straightforward element-wise alignment of intermediate features and prediction logits from ANNs and SNNs, often neglecting the intrinsic differences between their architectures. Specifically, ANN's outputs exhibit a continuous distribution, whereas SNN's outputs are characterized by sparsity and discreteness. To mitigate this issue, we introduce two innovative KD strategies. Firstly, we propose the Saliency-scaled Activation Map Distillation (SAMD), which aligns the spike activation map of the student SNN with the class-aware activation map of the teacher ANN. Rather than performing KD directly on the raw %and distinct features of ANN and SNN, our SAMD directs the student to learn from saliency activation maps that exhibit greater semantic and distribution consistency. Additionally, we propose a Noise-smoothed Logits Distillation (NLD), which utilizes Gaussian noise to smooth the sparse logits of student SNN, facilitating the alignment with continuous logits from teacher ANN. Extensive experiments on multiple datasets demonstrate the effectiveness of our methods. Code is available~\footnote{https://github.com/SinoLeu/CKDSNN.git}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。