arXiv:2501.05904cs.CV2025-01IJCAI被引 11

将变压器架构与脉冲神经网络结合,实现低功耗高效推理。

Binary Event-Driven Spiking Transformer

  • 用1比特权重和注意力图压缩模型,大幅降低存储与计算开销。
  • 在静态与类脑数据集上性能优于其他二值化脉冲神经网络。
  • 通过信息增强蒸馏缓解二值化导致的性能下降,适合边缘设备部署。

基于Transformer的脉冲神经网络(SNNs)引入了一种新型事件驱动的自注意力机制,结合了Transformer的高性能与SNN的高能效优势。然而,Transformer结构带来的更大模型尺寸和更高计算需求限制了其在资源受限场景中的实用性。本文将二值化技术融入基于Transformer的SNN,提出二值化事件驱动脉冲变压器(BESTformer)。该模型通过仅用1比特表示权重和注意力图,显著降低存储与计算需求。但其性能相比全精度模型有明显下降,主要因二值化表达能力有限。为此,我们提出耦合信息增强(CIE)方法,包含可逆框架与信息增强蒸馏。通过最大化二值模型与其全精度对应模型之间的互信息,有效缓解了性能退化问题。在静态及类脑数据集上的大量实验表明,该方法性能优于其他二值化SNN,展现出在资源受限边缘设备中兼具紧凑性与高性能的潜力。代码仓库地址:https://github.com/CaoHLin/BESTFormer。

原文摘要 · Abstract (English)

Transformer-based Spiking Neural Networks (SNNs) introduce a novel event-driven self-attention paradigm that combines the high performance of Transformers with the energy efficiency of SNNs. However, the larger model size and increased computational demands of the Transformer structure limit their practicality in resource-constrained scenarios. In this paper, we integrate binarization techniques into Transformer-based SNNs and propose the Binary Event-Driven Spiking Transformer, i.e. BESTformer. The proposed BESTformer can significantly reduce storage and computational demands by representing weights and attention maps with a mere 1-bit. However, BESTformer suffers from a severe performance drop from its full-precision counterpart due to the limited representation capability of binarization. To address this issue, we propose a Coupled Information Enhancement (CIE) method, which consists of a reversible framework and information enhancement distillation. By maximizing the mutual information between the binary model and its full-precision counterpart, the CIE method effectively mitigates the performance degradation of the BESTformer. Extensive experiments on static and neuromorphic datasets demonstrate that our method achieves superior performance to other binary SNNs, showcasing its potential as a compact yet high-performance model for resource-limited edge devices. The repository of this paper is available at https://github.com/CaoHLin/BESTFormer.

脉冲神经网络二值化边缘计算Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。