用分块替换法让脉冲网络学得更快更准
Efficient ANN-Guided Distillation: Aligning Rate-based Features of Spiking Neural Networks through Hybrid Block-wise Replacement
- 分块替换生成混合模型，逐步对齐脉冲网络与人工神经网络的特征
- 在多个数据集上达到或超越当前最优脉冲网络性能
- 适合想高效训练脉冲神经网络的研究者和工程师
脉冲神经网络（SNNs）作为人工神经网络（ANNs）的潜在替代方案受到广泛关注。近期研究展示了其在大规模数据集上的潜力。目前SNN训练主要分为直接训练和从ANN转换两种方式。为充分借助现有ANN模型指导SNN学习，可采用直接转换或ANN-SNN知识蒸馏方法。本文从ANN到SNN转换视角出发，提出一种基于分块替换策略的蒸馏框架，通过生成中间混合模型，逐步利用率编码特征对齐SNN与ANN的特征空间，自然引入率编码反向传播作为训练方法。该方法在训练效率和学习性能上均优于或相当于当前最优的SNN蒸馏方法。
原文摘要 · Abstract (English)
Spiking Neural Networks (SNNs) have garnered considerable attention as a potential alternative to Artificial Neural Networks (ANNs). Recent studies have highlighted SNNs' potential on large-scale datasets. For SNN training, two main approaches exist: direct training and ANN-to-SNN (ANN2SNN) conversion. To fully leverage existing ANN models in guiding SNN learning, either direct ANN-to-SNN conversion or ANN-SNN distillation training can be employed. In this paper, we propose an ANN-SNN distillation framework from the ANN-to-SNN perspective, designed with a block-wise replacement strategy for ANN-guided learning. By generating intermediate hybrid models that progressively align SNN feature spaces to those of ANN through rate-based features, our framework naturally incorporates rate-based backpropagation as a training method. Our approach achieves results comparable to or better than state-of-the-art SNN distillation methods, showing both training and learning efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。