无需训练即可将ANN转为高效脉冲神经网络，支持多种Transformer模型
Training-Free ANN-to-SNN Conversion for High-Performance Spiking Transformer
- 用多基指数衰减神经元逼近Transformer中的非线性运算
- 在视觉、自然语言任务中实现近无损精度转换，延迟显著降低
- 适合追求低功耗部署的AI系统开发者
利用事件驱动范式，脉冲神经网络（SNNs）为节能型Transformer架构提供了新路径。尽管将人工神经网络（ANN）转换为SNN可避免直接训练脉冲Transformer的高昂成本，但现有方法仍难以处理Transformer模块中的非线性操作，且常需对预训练ANN进行额外微调。为此，我们提出一种面向Transformer架构的免训练、高性能ANN-to-SNN转换框架。具体而言，引入多基指数衰减（MBE）神经元，结合指数衰减与多基编码策略，有效逼近非线性操作，无需修改预训练ANN的权重。在多种任务（计算机视觉、自然语言理解、自然语言生成）及主流Transformer模型（ViT、RoBERTa、GPT-2）上的大量实验表明，该方法实现了近无损转换精度，并显著降低延迟，为脉冲Transformer在真实场景中的高效可扩展部署提供了可行路径。
原文摘要 · Abstract (English)
Leveraging the event-driven paradigm, Spiking Neural Networks (SNNs) offer a promising approach for energy-efficient Transformer architectures.While ANN-to-SNN conversion avoids the high training cost of directly trained Spiking Transformers, existing approaches still struggle to handle the nonlinear operations within Transformer blocks, and often require additional fine-tuning of pretrained ANNs.To address these limitations, we propose a training-free and high-performance ANN-to-SNN conversion framework tailored for Transformer architectures. Specifically, we introduce a Multi-basis Exponential Decay (MBE) neuron that combines exponential decay with a multi-basis encoding strategy to effectively approximate nonlinear operations, eliminating the need for weight modifications in pretrained ANNs.Extensive experiments across diverse tasks (CV, NLU, NLG) and mainstream Transformer architectures (ViT, RoBERTa, GPT-2) demonstrate that our method achieves near-lossless conversion accuracy with significantly lower latency. This provides a promising pathway for the efficient and scalable deployment of Spiking Transformers in real-world applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。