arXiv:2605.20289cs.LGcs.AI2026-05中稿 · ICML

提出可插拔的脉冲非线性算子,让脉冲Transformer更高效运行。

Plug-and-Play Spiking Operators: Breaking the Nonlinearity Bottleneck in Spiking Transformers

论文配图:Plug-and-Play Spiking Operators: Breaking the Nonlinearity Bottleneck in Spiking Transformers
图 1 · 摘自论文原文
  • 将非线性运算拆解为除法、指数和ℓ2范数三类基础模块
  • 在不微调情况下替换后模型精度下降不足1%
  • 适合想部署脉冲大模型的硬件加速与神经形态计算研究者

人工神经网络到脉冲神经网络(ANN-to-SNN)转换为构建脉冲大语言模型提供了一条无需训练的实用路径。然而,现有流程主要关注Transformer中线性代数运算的脉冲实现,对关键非线性算子支持有限。这一缺口限制了与神经形态执行约束的兼容性,因为此类非线性通常需要除法、指数或范数计算,而这些操作无法自然由标准漏电积分发放(LIF)动力学支持。为此,我们提出一种可插拔框架,实现针对脉冲计算的非线性近似,并可集成至现有ANN-to-SNN流水线。该方法将非线性计算分解为三种常见基础单元——除法、指数和ℓ₂范数——并通过LIF神经元群组进行群体计算,结合轻量级位移缩放以避免浮点运算。通过组合这些基础模块,框架无需微调即可支持常见Transformer非线性(如Softmax、SiLU和归一化)。在多个LLM Transformer上的实验表明,选择性替换目标非线性算子带来的准确率下降均低于1%。

原文摘要 · Abstract (English)

ANN-to-SNN conversion offers a practical, training-free route to spiking large language models. However, current pipelines primarily focus on spike-driven realizations for Transformer linear-algebra operations, while providing limited support for key nonlinear operators. This gap limits compatibility with neuromorphic-style execution constraints, where such nonlinearities typically require division, exponentiation, or norm computations that are not naturally supported by standard leaky integrate-and-fire dynamics. To solve this problem, we propose a plug-and-play framework that implements spike-friendly approximations for Transformer nonlinearities and integrates into existing ANN-to-SNN pipelines. Our method decomposes these nonlinear computations into three recurring primitives -- division, exponentiation, and $\ell_2$ norms -- and realizes them via population computation using LIF neuron groups, combined with lightweight bit-shift scaling to avoid floating-point arithmetic. By composing these primitives as modular operator blocks, our framework supports common Transformer nonlinearities (e.g., Softmax, SiLU, and normalization) without any fine-tuning. Experiments on a range of LLMs Transformers show that selectively replacing the targeted nonlinear operators incurs less than a $1\%$ accuracy drop across all evaluated tasks.

脉冲神经网络大模型部署非线性算子神经形态计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。