通过时间分离与熵正则化提升脉冲神经网络的知识蒸馏效果
Temporal Separation with Entropy Regularization for Knowledge Distillation in Spiking Neural Networks
- 在不同时间步上进行对数蒸馏，捕捉脉冲网络的时序特性
- 引入熵正则化使训练更稳定，准确率显著提升
- 适合关注脉冲神经网络高效推理的科研与工程人员
受人脑启发的脉冲神经网络（SNN）通过离散脉冲传递信息，具备显著的计算效率优势。尽管能大幅降低推理能耗，但其性能仍落后于人工神经网络（ANN），主要受限于现有训练方法和模型本身。虽然近期研究尝试通过从ANN教师网络中进行知识蒸馏来提升SNN性能，但传统蒸馏方法常忽视SNN特有的时空特性，未能充分发挥其优势。为此，本文提出一种新型对数蒸馏方法，结合时间分离与熵正则化。该方法在不同时间步上对对数输出进行蒸馏，而非仅依赖聚合后的输出特征。同时，熵正则化有助于稳定优化过程，进一步提升性能。大量实验表明，本方法超越了现有的各类SNN蒸馏策略，包括基于对数蒸馏、特征蒸馏或二者结合的方法。代码将开源至GitHub。
原文摘要 · Abstract (English)
Spiking Neural Networks (SNNs), inspired by the human brain, offer significant computational efficiency through discrete spike-based information transfer. Despite their potential to reduce inference energy consumption, a performance gap persists between SNNs and Artificial Neural Networks (ANNs), primarily due to current training methods and inherent model limitations. While recent research has aimed to enhance SNN learning by employing knowledge distillation (KD) from ANN teacher networks, traditional distillation techniques often overlook the distinctive spatiotemporal properties of SNNs, thus failing to fully leverage their advantages. To overcome these challenge, we propose a novel logit distillation method characterized by temporal separation and entropy regularization. This approach improves existing SNN distillation techniques by performing distillation learning on logits across different time steps, rather than merely on aggregated output features. Furthermore, the integration of entropy regularization stabilizes model optimization and further boosts the performance. Extensive experimental results indicate that our method surpasses prior SNN distillation strategies, whether based on logit distillation, feature distillation, or a combination of both. The code will be available on GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。