arXiv:2502.14023cs.LGcs.AI2025-02被引 1

用知识蒸馏让脉冲神经网络动态激活,省电又保准。

Dynamic Activation with Knowledge Distillation for Energy-Efficient Spiking NN Ensembles

  • 教师网络指导多个学生脉冲网络,按功能拆分知识。
  • 只激活部分模型,比原模型省电20倍,精度仅降2%。
  • 在噪声下更稳定,适合低功耗设备部署。

尽管基础人工智能模型在分类和决策任务中表现优异,但其高能耗限制了在能源受限场景中的应用。受大脑效率启发,脉冲神经网络(SNN)因其事件驱动特性和与类脑芯片的兼容性,成为可行替代方案。本文提出一种结合知识蒸馏与集成学习的新系统:以基础AI模型为教师,指导一组小型学生SNN组成的脉冲神经集成(SNE)。SNE通过教师特征空间的有知划分实现知识解耦,使每个学生专注预测教师知识的特定方面,同时处理相同输入。核心创新在于基于知识蒸馏的自适应激活机制,仅动态启用部分学生模型,平衡精度与能效。该方法在CIFAR-10数据集上实现高达20倍的计算需求降低,精度仅下降2%;相比其他划分方式,准确率提升最高达2.4%。此外,面对噪声环境时,SNE表现出优于其ANN教师的鲁棒性。总体而言,SNE为能源受限应用提供了新方向。

原文摘要 · Abstract (English)

While foundation AI models excel at tasks like classification and decision-making, their high energy consumption makes them unsuitable for energy-constrained applications. Inspired by the brain's efficiency, spiking neural networks (SNNs) have emerged as a viable alternative due to their event-driven nature and compatibility with neuromorphic chips. This work introduces a novel system that combines knowledge distillation and ensemble learning to bridge the performance gap between artificial neural networks (ANNs) and SNNs. A foundation AI model acts as a teacher network, guiding smaller student SNNs organized into an ensemble, called Spiking Neural Ensemble (SNE). SNE enables the disentanglement of the teacher's knowledge, allowing each student to specialize in predicting a distinct aspect of it, while processing the same input. The core innovation of SNE is the adaptive activation of a subset of SNN models of an ensemble, leveraging knowledge-distillation, enhanced with an informed-partitioning (disentanglement) of the teacher's feature space. By dynamically activating only a subset of these student SNNs, the system balances accuracy and energy efficiency, achieving substantial energy savings with minimal accuracy loss. Moreover, SNE is significantly more efficient than the teacher network, reducing computational requirements by up to 20x with only a 2% drop in accuracy on the CIFAR-10 dataset. This disentanglement procedure achieves an accuracy improvement of up to 2.4% on the CIFAR-10 dataset compared to other partitioning schemes. Finally, we comparatively analyze SNE performance under noisy conditions, demonstrating enhanced robustness compared to its ANN teacher. In summary, SNE offers a promising new direction for energy-constrained applications.

脉冲神经网络知识蒸馏节能计算集成学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。