arXiv:2509.17461cs.CV2025-09被引 1

将传统Transformer转为全脉冲驱动模型,大幅降低训练成本与能耗。

CSDformer: A Conversion Method for Fully Spike-Driven Transformer

  • 设计新架构与NReLU函数替代softmax,适配脉冲神经网络转换
  • 在ImageNet上7步内达76.36%准确率,较现有方法更优
  • 无需训练脉冲网络,节省75%算力,提速2-3倍,适合低延迟场景

脉冲神经网络的Transformer架构旨在提升性能并缓解传统Transformer的高能耗问题。然而,现有生成方法存在显著缺陷:直接训练法成本过高,已有转换方法不可避免引入硬件不友好的操作。本文提出CSDformer,一种全新的全脉冲驱动Transformer转换方法。我们设计了面向转换的Transformer架构,并引入新函数NReLU替代自注意力中的softmax。该模型经量化与训练后,通过时间分解技术转化为全脉冲驱动模型。同时,提出延迟积分-放电神经元以降低转换误差,提升脉冲模型性能。在ImageNet、CIFAR-10和CIFAR-100数据集上评估,CSDformer在ImageNet上7个时间步实现76.36%的Top-1准确率,优于当前最优模型。此外,该方法无需训练SNN,计算资源减少75%,训练速度提升2-3倍。据我们所知,这是首个通过转换方式构建的全脉冲驱动Transformer模型,在超低延迟下实现高性能,同时显著降低计算复杂度与训练开销。

原文摘要 · Abstract (English)

Spike-based transformer is a novel architecture aiming to enhance the performance of spiking neural networks while mitigating the energy overhead inherent to transformers. However, methods for generating these models suffer from critical limitations: excessive training costs introduced by direct training methods, or unavoidably hardware-unfriendly operations in existing conversion methods. In this paper, we propose CSDformer, a novel conversion method for fully spike-driven transformers. We tailor a conversion-oriented transformer-based architecture and propose a new function NReLU to replace softmax in self-attention. Subsequently, this model is quantized and trained, and converted into a fully spike-driven model with temporal decomposition technique. Also, we propose delayed Integrate-andFire neurons to reduce conversion errors and improve the performance of spiking models. We evaluate CSDformer on ImageNet, CIFAR-10 and CIFAR-100 datasets and achieve 76.36% top-1 accuracy under 7 time-steps on ImageNet, demonstrating superiority over state-of-the-art models. Furthermore, CSDformer eliminates the need for training SNNs, thereby reducing training costs (reducing computational resource by 75% and accelerating training speed by 2-3$\times$). To the best of our knowledge, this is the first fully spike-driven transformer-based model developed via conversion method, achieving high performance under ultra-low latency, while dramatically reducing both computational complexity and training overhead.

脉冲神经网络Transformer低功耗模型转换

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。