针对脉冲神经网络训练数据缺失问题,提出新方法提升模型能效与性能。
STARS: Spike Tail-Aware Relational Synthesis for ANN-to-SNN Data-Free Knowledge Distillation

- 通过关系一致性与尾部概率约束,生成更适配脉冲神经网络的合成数据
- 在CIFAR-10和CIFAR-100上分别提升4.6%和6.7%准确率
- 适用于无原始数据场景下的脉冲神经网络知识蒸馏,尤其适合低功耗部署
脉冲神经网络(SNN)具有低功耗、低延迟推理潜力,但性能仍落后于人工神经网络(ANN)。ANN-to-SNN知识蒸馏有助于缩小这一差距,但在实际部署中常缺乏原始训练数据。现有数据无关知识蒸馏(DFKD)方法通过匹配教师模型的先验信息(如批归一化统计量)合成代理数据,但这些面向ANN的约束主要规范均值与方差,对依赖阈值穿越动态的SNN学生模型而言仍约束不足。本文提出脉冲尾部感知关系合成(STARS),一种即插即用的ANN-to-SNN DFKD方法,通过补充两个互补目标:关系一致性对齐(保持师生间样本间关系一致性)与尾部感知正则化(通过软超限方式正则化教师导出阈值相关的尾部概率)。二者协同生成既符合教师有效性的合成批次,又对SNN学生更具信息性。在多个ANN-SNN组合下的CIFAR-10、CIFAR-100和Tiny-ImageNet实验表明,该方法持续优于传统DFKD基线,甚至超越部分其他蒸馏方法,在CIFAR-10上最高提升4.6%,在CIFAR-100上达6.7%,凸显在面向SNN的DFKD中,补充关系与尾部感知约束的重要性。
原文摘要 · Abstract (English)
SNNs promise energy-efficient and low-latency inference, but their performance still trails that of ANNs. ANN-to-SNN knowledge distillation helps narrow this gap, yet the original training data are often unavailable in practical deployment settings. Existing data-free knowledge distillation (DFKD) methods synthesize surrogate data by matching teacher-side priors, especially BN statistics, but these ANN-oriented constraints mainly regularize mean and variance and therefore remain under-constrained for SNN students whose responses depend on threshold-crossing dynamics. In this paper, we propose Spike Tail-Aware Relational Synthesis (STARS), a plug-and-play method for ANN-to-SNN DFKD that augments standard BN-guided synthesis with two complementary objectives: Relational Consistency Alignment, which preserves cross-sample relational consistency between teacher and student, and Tail-Aware Regularization, which regularizes threshold-relevant tail probabilities through soft exceedance over teacher-derived thresholds. Together, these objectives generate synthetic batches that remain teacher-valid while becoming more informative for SNN students. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet across multiple ANN-SNN pairs show that our method consistently improves conventional DFKD baselines and even surpasses several KD methods, with gains of up to 4.6\% on CIFAR-10 and 6.7\% on CIFAR-100, highlighting the importance of complementing BN matching with relational and tail-aware constraints in SNN-oriented DFKD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。