为脉冲神经网络设计自适应信任域,提升无梯度优化的稳定性与效率。
Signal-Adaptive Trust Regions for Gradient-Free Optimization of Recurrent Spiking Neural Networks
- 基于信号能量自适应调节更新范围,抑制噪声干扰
- 在有限种群下实现更稳定训练,性能媲美PPO-LSTM
- 支持二值化脉冲与权重,显著加速大规模训练
循环脉冲神经网络(RSNN)是实现低功耗控制策略的有前景方案,但其在高维、长时程强化学习中的训练仍具挑战。群体式无梯度优化通过估计梯度绕过不可微的脉冲动力学,但有限种群导致估计方差大,引发有害且激进的更新。受强化学习中信任区域方法启发,我们提出信号自适应信任区域(SATR),一种在分布空间约束相对变化的更新规则,通过归一化信号能量的KL散度限制更新幅度。SATR在强信号时自动扩大信任域,在噪声主导时收缩。我们将SATR应用于伯努利连接分布,该分布已在RSNN优化中表现优异。在多个高维连续控制基准测试中,SATR在种群受限时提升训练稳定性,并达到与PPO-LSTM等强基线相当的回报。此外,为实现可扩展性,我们引入位集实现二值脉冲与二值权重,大幅降低训练时间,支持快速政策搜索。
原文摘要 · Abstract (English)
Recurrent spiking neural networks (RSNNs) are a promising substrate for energy-efficient control policies, but training them for high-dimensional, long-horizon reinforcement learning remains challenging. Population-based, gradient-free optimization circumvents backpropagation through non-differentiable spike dynamics by estimating gradients. However, with finite populations, high variance of these estimates can induce harmful and overly aggressive update steps. Inspired by trust-region methods in reinforcement learning that constrain policy updates in distribution space, we propose \textbf{Signal-Adaptive Trust Regions (SATR)}, a distributional update rule that constrains relative change by bounding KL divergence normalized by an estimated signal energy. SATR automatically expands the trust region under strong signals and contracts it when updates are noise-dominated. We instantiate SATR for Bernoulli connectivity distributions, which have shown strong empirical performance for RSNN optimization. Across a suite of high-dimensional continuous-control benchmarks, SATR improves stability under limited populations and reaches competitive returns against strong baselines including PPO-LSTM. In addition, to make SATR practical at scale, we introduce a bitset implementation for binary spiking and binary weights, substantially reducing wall-clock training time and enabling fast RSNN policy search.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。