让脉冲神经网络的视觉变压器更省电,还能提准。
STAS: Spatio-Temporal Adaptive Computation Time for Spiking Transformers
- 设计新架构与动态计算协同优化,解决脉冲网络延迟高问题。
- 在三个图像数据集上,能耗最高降45.9%,准确率还超现有模型。
- 适合做低功耗智能视觉系统的研究者和工程师参考。
脉冲神经网络(SNN)相比人工神经网络(ANN)具有更高能效,但因其多时间步运行机制,存在高延迟和计算开销问题。尽管已有多种动态计算方法针对空间、时间或结构冗余进行优化,但这些方法仍零散分散。自适应计算时间(ACT)原理为统一方法提供了坚实基础,但其应用于基于脉冲神经网络的视觉变压器(ViTs)时面临两大核心挑战:违反时间相似性前提,且静态架构与该原则不兼容。为此,我们提出STAS(用于脉冲变压器的时空自适应计算时间),通过协同设计静态架构与动态计算策略来解决上述问题。STAS引入集成脉冲块分割(I-SPS)模块,通过创建统一输入表示实现时间稳定性,从而解决时间不一致的架构问题。这一稳定性使我们的自适应脉冲自注意力(A-SSA)模块能够在空间和时间两个维度上同时进行标记剪枝。在脉冲变压器架构上实现,并在CIFAR-10、CIFAR-100和ImageNet上验证,STAS分别将能耗降低45.9%、43.8%和30.1%,同时提升准确率超过当前最优模型。
原文摘要 · Abstract (English)
Spiking neural networks (SNNs) offer energy efficiency over artificial neural networks (ANNs) but suffer from high latency and computational overhead due to their multi-timestep operational nature. While various dynamic computation methods have been developed to mitigate this by targeting spatial, temporal, or architecture-specific redundancies, they remain fragmented. While the principles of adaptive computation time (ACT) offer a robust foundation for a unified approach, its application to SNN-based vision Transformers (ViTs) is hindered by two core issues: the violation of its temporal similarity prerequisite and a static architecture fundamentally unsuited for its principles. To address these challenges, we propose STAS (Spatio-Temporal Adaptive computation time for Spiking transformers), a framework that co-designs the static architecture and dynamic computation policy. STAS introduces an integrated spike patch splitting (I-SPS) module to establish temporal stability by creating a unified input representation, thereby solving the architectural problem of temporal dissimilarity. This stability, in turn, allows our adaptive spiking self-attention (A-SSA) module to perform two-dimensional token pruning across both spatial and temporal axes. Implemented on spiking Transformer architectures and validated on CIFAR-10, CIFAR-100, and ImageNet, STAS reduces energy consumption by up to 45.9%, 43.8%, and 30.1%, respectively, while simultaneously improving accuracy over SOTA models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。