arXiv:2608.19238cs.NEcs.CV2026-08

提升脉冲变压器的局部交互与融合能力,增强模型对空间关系的捕捉。

Spiking Local Interaction and Adaptive Complementary Fusion for Spiking Transformer

论文配图:Spiking Local Interaction and Adaptive Complementary Fusion for Spiking Transformer
图 1 · 摘自论文原文
  • 引入独立的局部脉冲交互路径,实现邻近神经元直接通信
  • 通过可学习融合机制,使自注意力与局部交互互补,提升性能
  • 适用于图像分类、事件视觉和语义分割,无需预训练即有效

脉冲变压器主要依赖脉冲自注意力(SSA)建模标记间交互。然而,二值化的查询与键将连续相似性映射为稀疏离散的关系响应,可能抑制弱关系并限制局部空间上下文传播。为此,本文提出脉冲局部交互(SLI)与自适应互补融合(ACF)。SLI通过轻量级深度-点卷积构建独立于注意力的邻近脉冲标记间信息交换路径。ACF利用逐层、逐通道的可学习系数,在不同网络深度上自适应平衡SSA与SLI的贡献。该设计保持原始注意力形式,参数增量小,可集成至多种脉冲变压器架构。在ImageNet-1K、CIFAR-10、CIFAR-100、CIFAR10-DVS和ADE20K上的实验表明,各类任务均取得一致提升。其中,采用SLI与ACF的QKFormer在ImageNet-1K上达到84.37% Top-1准确率,在ADE20K上达37.5% mIoU,且分割模型未使用ImageNet预训练。消融研究与定性分析表明,SSA与SLI捕获互补的交互模式,可学习融合优于固定加权。

原文摘要 · Abstract (English)

Spiking Transformers model token interactions primarily through spiking self-attention (SSA). However, binary query and key representations map continuous similarities to sparse and discrete relation responses, which may suppress weak relations and limit the propagation of local spatial context. To address this limitation, we introduce Spiking Local Interaction (SLI) and Adaptive Complementary Fusion (ACF). SLI establishes an attention-independent pathway for direct information exchange among neighboring spiking tokens using lightweight depthwise--pointwise transformations. ACF integrates SSA and SLI through layer-specific, channel-wise coefficients that adaptively balance their contributions at different network depths. The proposed design preserves the original attention formulation and can be incorporated into different Spiking Transformer architectures with modest parameter overhead. Experiments on ImageNet-1K, CIFAR-10, CIFAR-100, CIFAR10-DVS, and ADE20K show consistent improvements across image classification, event-based recognition, and semantic segmentation. In particular, QKFormer with SLI and ACF achieves $84.37\%$ Top-1 accuracy on ImageNet-1K and $37.5\%$ mIoU on ADE20K, where the segmentation model is trained without ImageNet pretraining. Ablation studies and qualitative analyses further indicate that SSA and SLI capture complementary interaction patterns and that learnable fusion consistently outperforms fixed weighting.

脉冲神经网络注意力机制图像分割事件相机

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。