用贝叶斯方法训练二值和脉冲神经网络，无需归一化层也能达顶尖性能。
A Principled Bayesian Framework for Training Binary and Spiking Neural Networks
- 基于概率模型设计无偏梯度估计器，实现端到端优化
- 在CIFAR-10等数据集上性能超越或媲美现有方法
- 适合需要低功耗、高效率的神经网络部署场景
我们提出一种贝叶斯框架，用于训练二值与脉冲神经网络，无需归一化层即可达到当前最优性能。与通常依赖启发式超参数的代理梯度方法不同，本方法基于噪声二值网络的概率模型，支持全端到端的梯度优化。引入重要性加权直通（IW-ST）估计器，统一了直通与松弛类估计器；分析其偏差-方差权衡，并通过辅助损失实现最小偏差目标。在此基础上，提出脉冲贝叶斯神经网络（SBNNs），利用后验噪声进行变分推断，以IW-ST训练二值与脉冲网络。该方法降低梯度偏差、正则化参数并引入类似丢弃的噪声。通过关联低偏差条件、梯度消失与KL项，实现无需归一化的深层残差网络训练。在CIFAR-10、DVS Gesture和SHD数据集上的实验表明，本方法无需归一化或手工调参梯度即可匹配或超越现有方法。
原文摘要 · Abstract (English)
We propose a Bayesian framework for training binary and spiking neural networks that achieves state-of-the-art performance without normalisation layers. Unlike commonly used surrogate gradient methods -- often heuristic and sensitive to hyperparameter choices -- our approach is grounded in a probabilistic model of noisy binary networks, enabling fully end-to-end gradient-based optimisation. We introduce importance-weighted straight-through (IW-ST) estimators, a unified class generalising straight-through and relaxation-based estimators. We characterise the bias-variance trade-off in this family and derive a bias-minimising objective implemented via an auxiliary loss. Building on this, we introduce Spiking Bayesian Neural Networks (SBNNs), a variational inference framework that uses posterior noise to train Binary and Spiking Neural Networks with IW-ST. This Bayesian approach minimises gradient bias, regularises parameters, and introduces dropout-like noise. By linking low-bias conditions, vanishing gradients, and the KL term, we enable training of deep residual networks without normalisation. Experiments on CIFAR-10, DVS Gesture, and SHD show our method matches or exceeds existing approaches without normalisation or hand-tuned gradients.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。