arXiv:2504.20445cs.AI2025-04中稿 · IJCNN2025被引 2

针对脉冲神经网络知识蒸馏中忽略低概率预测的问题,提出新方法提升性能。

Head-Tail-Aware KL Divergence in Knowledge Distillation for Spiking Neural Networks

  • 设计基于累积概率的动态掩码,区分高低概率区域
  • 融合前向与反向KL散度,在少时步下性能超越现有方法
  • 特别适合资源受限场景下的脉冲神经网络训练

脉冲神经网络(SNNs)因其能效高和生物合理性而备受关注,但其训练方法和模型约束导致与人工神经网络(ANNs)存在性能差距。知识蒸馏(KD)被用于将ANN教师模型的知识迁移至SNN学生模型以缓解此问题。传统基于KL散度的方法往往过度关注高概率预测,忽略低概率部分,影响泛化能力。为此,本文提出头尾感知的KL散度(HTA-KL),通过累积概率构建动态掩码,自适应加权以平衡高低概率区域的知识传递。结合前向KL(FKL)与反向KL(RKL)散度,有效对齐分布的头部与尾部。在CIFAR-10、CIFAR-100和Tiny ImageNet数据集上验证,本方法在多数情况下以更少的时步实现优于现有方法的性能。

原文摘要 · Abstract (English)

Spiking Neural Networks (SNNs) have emerged as a promising approach for energy-efficient and biologically plausible computation. However, due to limitations in existing training methods and inherent model constraints, SNNs often exhibit a performance gap when compared to Artificial Neural Networks (ANNs). Knowledge distillation (KD) has been explored as a technique to transfer knowledge from ANN teacher models to SNN student models to mitigate this gap. Traditional KD methods typically use Kullback-Leibler (KL) divergence to align output distributions. However, conventional KL-based approaches fail to fully exploit the unique characteristics of SNNs, as they tend to overemphasize high-probability predictions while neglecting low-probability ones, leading to suboptimal generalization. To address this, we propose Head-Tail Aware Kullback-Leibler (HTA-KL) divergence, a novel KD method for SNNs. HTA-KL introduces a cumulative probability-based mask to dynamically distinguish between high- and low-probability regions. It assigns adaptive weights to ensure balanced knowledge transfer, enhancing the overall performance. By integrating forward KL (FKL) and reverse KL (RKL) divergence, our method effectively align both head and tail regions of the distribution. We evaluate our methods on CIFAR-10, CIFAR-100 and Tiny ImageNet datasets. Our method outperforms existing methods on most datasets with fewer timesteps.

知识蒸馏脉冲神经网络分布对齐高效计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。