arXiv:2608.08541cs.CVcs.NE2026-08

提出新型脉冲注意力机制,显著提升模型空间局部性与性能。

Rethinking Attention Locality in Spiking Transformers

论文配图:Rethinking Attention Locality in Spiking Transformers
图 1 · 摘自论文原文
  • 在不重叠区域计算注意力,通过轻量卷积路径跨边界传递信息。
  • 在七个数据集上实现最高9.50%的mAP@50和3.42%的mIoU提升。
  • 适配主流脉冲变压器架构,参数与能耗增长极少。

脉冲变压器以脉冲驱动计算为视觉处理提供高效范式,但其无Softmax的脉冲自注意力(SSA)难以建立空间局部的标记交互。尽管现有局部增强方法提升准确率,但尚不清楚它们是否在各层及不同架构中一致诱导空间局部性。通过平均注意力距离(MAD)分析,我们发现计算局部性不必然带来空间局部性,并指出统一应用同一增强策略忽略了架构依赖的部署需求。为此,我们提出具有边界连续路径的空间连续局部注意力(SCLA-BCP)。SCLA在空间相邻标记的非重叠区域计算注意力,而BCP通过轻量卷积路径促进跨边界信息交换。此外,我们设计了分层局部性部署策略,有效应用于两类主流脉冲变压器架构。在涵盖分类、检测与分割的七个静态及类脑数据集上的实验表明,该方法在极少增加参数与能量开销下实现持续提升。尤其在COCO 2017上mAP@50提升达9.50%,在ADE20K上mIoU提升达3.42%。可视化、MAD分析与消融实验进一步验证其有效性。

原文摘要 · Abstract (English)

Spiking Transformers provide a promising paradigm for efficient visual processing with spike-driven computation, yet their Softmax-free Spiking Self-Attention (SSA) struggles to establish spatially localized token interactions. Although existing locality-enhanced SSA methods improve accuracy, it remains unclear whether they consistently induce spatial locality across layers and different Spiking Transformer architectures. Through Mean Attention Distance (MAD) analysis, we reveal that computational locality does not necessarily translate into spatial locality and show that uniformly applying the same locality enhancement overlooks architecture-dependent deployment requirements. Motivated by these observations, we propose Spatially Contiguous Local Attention with Boundary Continuity Pathway (SCLA-BCP). SCLA computes attention within non-overlapping regions of spatially adjacent tokens, while BCP facilitates cross-boundary information exchange through a lightweight convolutional pathway. Furthermore, we develop a hierarchical locality deployment strategy to effectively apply SCLA-BCP across the two major Spiking Transformer architectures. Extensive experiments on seven static and neuromorphic datasets covering classification, detection, and segmentation demonstrate consistent improvements with limited parameter and energy overhead. Notably, our approach improves mAP@50 by up to 9.50% on COCO 2017 and mIoU by up to 3.42% on ADE20K. Visualizations, MAD analysis, and ablation studies further validate its effectiveness.

脉冲神经网络注意力机制类脑计算目标检测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。