首个面向离线强化学习的脉冲驱动变压器,节能78.4%且性能不降。
Decision SpikeFormer: Spike-Driven Transformer for Decision Making
- 设计脉冲驱动的时序自注意力机制,捕捉决策序列中的时间依赖性。
- 在D4RL基准上实现78.4%能耗降低,性能优于传统神经网络与脉冲网络。
- 适合低功耗智能体决策场景,如嵌入式AI与边缘设备应用。
离线强化学习(Offline RL)仅依靠预采集数据训练策略,避免与环境直接交互,对能源受限的具身智能应用至关重要。尽管基于人工神经网络(ANN)的方法在离线RL中表现良好,但其高计算与能耗需求促使探索更高效的替代方案。脉冲神经网络(SNN)因低功耗特性展现出潜力。本文提出DSFormer,首个专为离线RL设计的脉冲驱动变压器模型,通过序列建模实现决策。不同于聚焦视觉空间维度的现有SNN Transformer,我们设计了时序脉冲自注意力(TSSA)和位置脉冲自注意力(PSSA),以捕捉强化学习中序列建模所需的时间与位置依赖关系。此外,提出渐进阈值依赖批量归一化(PTBN),结合层归一化与批量归一化的优点,在保持脉冲特性的同时维持时间依赖性。在D4RL基准上的全面实验表明,DSFormer在能耗上优于同类SNN与ANN模型,实现78.4%的能效提升,凸显其在能效与性能上的双重优势。代码与模型已公开于 https://wei-nijuan.github.io/DecisionSpikeFormer。
原文摘要 · Abstract (English)
Offline reinforcement learning (RL) enables policy training solely on pre-collected data, avoiding direct environment interaction - a crucial benefit for energy-constrained embodied AI applications. Although Artificial Neural Networks (ANN)-based methods perform well in offline RL, their high computational and energy demands motivate exploration of more efficient alternatives. Spiking Neural Networks (SNNs) show promise for such tasks, given their low power consumption. In this work, we introduce DSFormer, the first spike-driven transformer model designed to tackle offline RL via sequence modeling. Unlike existing SNN transformers focused on spatial dimensions for vision tasks, we develop Temporal Spiking Self-Attention (TSSA) and Positional Spiking Self-Attention (PSSA) in DSFormer to capture the temporal and positional dependencies essential for sequence modeling in RL. Additionally, we propose Progressive Threshold-dependent Batch Normalization (PTBN), which combines the benefits of LayerNorm and BatchNorm to preserve temporal dependencies while maintaining the spiking nature of SNNs. Comprehensive results in the D4RL benchmark show DSFormer's superiority over both SNN and ANN counterparts, achieving 78.4% energy savings, highlighting DSFormer's advantages not only in energy efficiency but also in competitive performance. Code and models are public at https://wei-nijuan.github.io/DecisionSpikeFormer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。