提出一种高效稀疏的脉冲状态空间模型,解决长序列学习中的能效与精度难题。
SPikE-SSM: A Sparse, Precise, and Efficient Spiking State Space Model for Long Sequences Learning
- 通过边界压缩策略实现脉冲神经元并行推理,提升长序列处理效率。
- 设计具生物可解释性的脉冲神经元模型,支持动态时间计算。
- 通过可训练阈值和抑制强度平衡稀疏性与精度,适合低功耗场景。
脉冲神经网络(SNNs)利用生物系统的脉冲特性和稀疏性,提供节能解决方案。尽管自Transformer出现以来,SNNs在长序列任务上难以与人工网络竞争,但状态空间模型(SSMs)的兴起带来了更高的计算效率和建模能力。然而,将高效的SSMs应用于SNNs进行长序列学习面临三大挑战:(1)膜电位依赖于神经元过去的脉冲历史,导致并行计算下序列建模效率降低;(2)生物脉冲神经元复杂的动力学特性对功能至关重要,却难以在大规模网络中有效模拟与利用;(3)在不采用密集计算的前提下,难以在保持高稀疏性的同时实现高精度。为此,我们提出一种稀疏、精确且高效的脉冲状态空间模型框架——SPikE-SSM。针对(1),提出边界压缩策略(PMBC),加速脉冲神经元推断,支持长序列并行学习;针对(2),设计一种新颖简洁的神经元模型,融合复位-不应期机制,利用内在的时间维度实现动态计算并具备生物可解释性;针对(3),将所提神经元模型分层集成至原SSM模块,并引入可训练的阈值和不应期强度,以平衡精度与稀疏性。大量实验验证了SPikE-SSM在长程基准和大语言数据集WikiText-103上的有效性与鲁棒性,展示了动态脉冲神经元在高效长序列学习中的潜力。
原文摘要 · Abstract (English)
Spiking neural networks (SNNs) provide an energy-efficient solution by utilizing the spike-based and sparse nature of biological systems. Since the advent of Transformers, SNNs have struggled to compete with artificial networks on long sequential tasks, until the recent emergence of state space models (SSMs), which offer superior computational efficiency and modeling capability. However, applying the highly capable SSMs to SNNs for long sequences learning poses three major challenges: (1) The membrane potential is determined by the past spiking history of the neuron, leading to reduced efficiency for sequence modeling in parallel computing scenarios. (2) Complex dynamics of biological spiking neurons are crucial for functionality but challenging to simulate and exploit effectively in large networks. (3) It is arduous to maintain high sparsity while achieving high accuracy for spiking neurons without resorting to dense computing, as utilized in artificial neuron-based SSMs. To address them, we propose a sparse, precise and efficient spiking SSM framework, termed SPikE-SSM. For (1), we propose a boundary compression strategy (PMBC) to accelerate the inference of the spiking neuron model, enabling parallel processing for long sequence learning. For (2), we propose a novel and concise neuron model incorporating reset-refractory mechanism to leverage the inherent temporal dimension for dynamic computing with biological interpretability. For (3), we hierarchically integrate the proposed neuron model to the original SSM block, and enhance the dynamics of SPikE-SSM by incorporating trainable thresholds and refractory magnitudes to balance accuracy and sparsity. Extensive experiments verify the effectiveness and robustness of SPikE-SSM on the long range arena benchmarks and large language dataset WikiText-103, showing the potential of dynamic spiking neurons in efficient long sequence learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。