用侧抑制机制提升脉冲注意力，让模型更专注关键信息
SpiLiFormer: Enhancing Spiking Transformers with Lateral Inhibition
- 借鉴大脑侧抑制机制，动态调节脉冲注意力分布
- 在多个数据集上达到新SOTA，ImageNet精度领先0.46%
- 参数量仅为对手39%，时间步数减半，适合边缘部署
基于Transformer的脉冲神经网络（SNN）因其优异性能和高能效受到关注。然而，现有基于Transformer的SNN中的脉冲注意力模块多直接沿用模拟Transformer设计，未能充分解决对无关上下文过度分配注意力的问题。为此，我们提出受侧抑制启发的脉冲Transformer（SpiLiFormer），模拟大脑的侧抑制机制，引导模型增强对相关标记的关注，同时抑制对无关标记的注意力。该模型在多个数据集上达到最新性能：CIFAR-10（+0.45%）、CIFAR-100（+0.48%）、CIFAR10-DVS（+2.70%）、N-Caltech101（+1.94%）、ImageNet-1K（+1.6%）。特别地，在ImageNet-1K上，SpiLiFormer（6990万参数，4个时间步，384分辨率）以仅39%的参数量和一半的时间步数，超越E-SpikeFormer（1.73亿参数，8个时间步，384分辨率），精度领先0.46%。代码与模型权重已公开于https://github.com/KirinZheng/SpiLiFormer。
原文摘要 · Abstract (English)
Spiking Neural Networks (SNNs) based on Transformers have garnered significant attention due to their superior performance and high energy efficiency. However, the spiking attention modules of most existing Transformer-based SNNs are adapted from those of analog Transformers, failing to fully address the issue of over-allocating attention to irrelevant contexts. To fix this fundamental yet overlooked issue, we propose a Lateral Inhibition-inspired Spiking Transformer (SpiLiFormer). It emulates the brain's lateral inhibition mechanism, guiding the model to enhance attention to relevant tokens while suppressing attention to irrelevant ones. Our model achieves state-of-the-art (SOTA) performance across multiple datasets, including CIFAR-10 (+0.45%), CIFAR-100 (+0.48%), CIFAR10-DVS (+2.70%), N-Caltech101 (+1.94%), and ImageNet-1K (+1.6%). Notably, on the ImageNet-1K dataset, SpiLiFormer (69.9M parameters, 4 time steps, 384 resolution) outperforms E-SpikeFormer (173.0M parameters, 8 time steps, 384 resolution), a SOTA spiking Transformer, by 0.46% using only 39% of the parameters and half the time steps. The code and model checkpoints are publicly available at https://github.com/KirinZheng/SpiLiFormer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。