提出首个无需梯度的全二值多层神经网络训练方法,大幅降低计算开销。
Training Multi-Layer Binary Neural Networks With Local Binary Error Signals
- 用局部二值误差信号和二值权重更新实现无梯度训练
- 在相同内存下比全精度SGD提升35.30%准确率,计算量降2-3个数量级
- 适合资源受限场景,如边缘设备部署
二值神经网络(BNNs)通过将权重和激活仅用1比特表示,显著降低计算复杂度和内存占用。然而,现有BNN训练算法大多依赖量化感知的浮点随机梯度下降(SGD),仅能在推理阶段发挥二值操作优势。本文首次提出一种完全二值且无需梯度的多层BNN训练算法,彻底消除对反向传播浮点梯度的依赖。该方法基于局部二值误差信号与二值权重更新,采用整数隐含权重作为突触元塑性机制,增强神经生物学合理性。所提方案仅需XNOR、Popcount及增减操作即可训练二值多层感知机。在多分类基准测试中,相比唯一现有全二值单层最优方案,测试准确率最高提升35.47%;相较全精度SGD,在相同总内存需求下,准确率最高提升35.30%,同时计算成本降低2至3个数量级(以布尔门总数计)。该算法已公开于科学社区开源仓库。
原文摘要 · Abstract (English)
Binary Neural Networks (BNNs) significantly reduce computational complexity and memory usage in machine and deep learning by representing weights and activations with just one bit. However, most existing training algorithms for BNNs rely on quantization-aware floating-point Stochastic Gradient Descent (SGD), limiting the full exploitation of binary operations to the inference phase only. In this work, we propose, for the first time, a fully binary and gradient-free training algorithm for multi-layer BNNs, eliminating the need for back-propagated floating-point gradients. Specifically, the proposed algorithm relies on local binary error signals and binary weight updates, employing integer-valued hidden weights that serve as a synaptic metaplasticity mechanism, thereby enhancing its neurobiological plausibility. Our proposed solution enables the training of binary multi-layer perceptrons by using exclusively XNOR, Popcount, and increment/decrement operations. Experimental results on multi-class classification benchmarks show test accuracy improvements of up to +35.47% over the only existing fully binary single-layer state-of-the-art solution. Compared to full-precision SGD, our solution improves test accuracy by up to +35.30% under the same total memory demand, while also reducing computational cost by two to three orders of magnitude in terms of the total number of Boolean gates. The proposed algorithm is made available to the scientific community as a public repository.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。