用注意力熵动态调整脉冲网络梯度，提升模型训练效果。
SAGE: Surrogate-gradient Adaptation via Attention-Guided Entropy for Spiking Transformers
- 根据注意力熵估计不确定性，动态调节梯度斜率。
- 在CIFAR-10/100上比固定梯度方法提升1-2%准确率。
- 无需修改推理结构，适合部署在资源受限设备。
脉冲神经网络（SNN）通过稀疏事件驱动计算实现能效优势，但其训练困难，因非可微的脉冲函数需依赖代理梯度，而固定形状的代理梯度在不同层和训练阶段可能不最优。本文提出SAGE，一种面向基于Transformer的SNN的不确定性调制代理梯度机制。SAGE通过归一化自注意力熵估计块级不确定性，并利用该信号在训练中动态调整代理梯度斜率，推理阶段保持原模型不变。仅调节训练时的代理参数，保留原始架构与部署开销，同时增强优化灵活性。在CIFAR-10/100上的实验表明，SAGE相较固定代理梯度基线获得显著性能提升，多个仿真时间步下一致取得1-2%的准确率增益。结果表明，注意力导出的不确定性可作为轻量级训练信号，用于Transformer-based SNN的自适应代理梯度学习。
原文摘要 · Abstract (English)
Spiking neural networks (SNNs) offer an energy-efficient alternative to conventional deep neural networks by exploiting sparse event-driven computation, but their training remains challenging because the non-differentiable spike function requires surrogate gradients whose fixed shape may be suboptimal across layers and training stages. In this work, we introduce SAGE, an uncertainty-modulated surrogate-gradient mechanism for Transformer-based SNNs. SAGE estimates block-level uncertainty from normalized self-attention entropy and uses this signal to adapt the surrogate-gradient slope during training while leaving the inference model unchanged. By modulating only the training-time surrogate parameter, the proposed method preserves the original architecture and deployment cost while improving optimization flexibility. Experiments on CIFAR-10/100 demonstrate that SAGE achieves improved accuracy over fixed-surrogate baselines, with results up to 1-2\% consistent gains across multiple simulation time steps. These results highlight the potential of attention-derived uncertainty as a lightweight training signal for adaptive surrogate-gradient learning in transformer-based SNNs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。