arXiv:2508.01992cs.LGq-bio.NC2025-08

通过剪枝与协同学习,让脉冲Transformer更轻更快。

Toward Efficient Spiking Transformers: Synapse Pruning Meets Synergistic Learning-Based Compensation

  • 用无结构L1P和有结构DSP剪枝减少权重冗余。
  • 模型参数量大幅下降,计算开销降低但性能仍优。
  • 适合部署在资源受限设备的高效脉冲神经网络设计。

作为人工智能模型的基础架构,Transformer最近被适配到脉冲神经网络,在多种任务中展现出良好性能。然而,现有的脉冲Transformer(ST)模型参数量庞大,计算成本高,限制了其在资源受限环境中的部署。为解决这一问题,本文提出将突触剪枝与协同学习补偿策略结合,构建轻量化ST模型。具体地,引入两种定制化剪枝策略:无结构L1P方法用于生成稀疏表示,结构化DSP方法用于生成低秩表示,以减少ST模块权重矩阵中的冗余。此外,提出一种增强型脉冲神经元模型——协同漏电整合-放电(sLIF)神经元,通过突触可塑性与内在可塑性的协同学习机制,有效补偿剪枝带来的性能损失。在基准数据集上的大量实验表明,所提方法显著降低了模型规模和计算开销,同时保持了竞争力的性能。结果验证了所提剪枝与补偿策略在构建高效且高性能的ST模型中的有效性。

原文摘要 · Abstract (English)

As a foundational architecture of artificial intelligence models, Transformer has been recently adapted to spiking neural networks with promising performance across various tasks. However, existing spiking Transformer(ST)-based models require a substantial number of parameters and incur high computational costs, thus limiting their deployment in resource-constrained environments. To address these challenges, we propose combining synapse pruning with a synergistic learning-based compensation strategy to derive lightweight ST-based models. Specifically, two types of tailored pruning strategies are introduced to reduce redundancy in the weight matrices of ST blocks: an unstructured $\mathrm{L_{1}P}$ method to induce sparse representations, and a structured DSP method to induce low-rank representations. In addition, we propose an enhanced spiking neuron model, termed the synergistic leaky integrate-and-fire (sLIF) neuron, to effectively compensate for model pruning through synergistic learning between synaptic and intrinsic plasticity mechanisms. Extensive experiments on benchmark datasets demonstrate that the proposed methods significantly reduce model size and computational overhead while maintaining competitive performance. These results validate the effectiveness of the proposed pruning and compensation strategies in constructing efficient and high-performing ST-based models.

脉冲神经网络模型剪枝协同学习轻量化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。