提出首个全二值反向传播算法,实现高效二值神经网络端到端训练。
BEP: A Binary Error Propagation Algorithm for Binary Neural Networks Training
- 基于二值变量的链式法则,用位运算实现误差反向传播
- 多层感知机和循环网络测试准确率提升最高达+10.57%
- 无需浮点运算,适合资源受限设备部署
二值神经网络(BNNs)将权重和激活均限制为二值,显著降低计算复杂度、内存占用和能耗,特别适用于资源受限设备。然而,由于变量离散性,基于梯度的训练仍具挑战。主流方法量化感知训练需保留全精度参数并使用浮点反向传播,牺牲了训练时的二值效率。现有基于局部学习规则的方法不适用于全局信用分配及多层架构的误差反传。本文提出二值误差传播(BEP),首个建立反向传播链式法则的离散化方法,使误差信号以二值向量形式在多层网络中反向传播。BEP完全在二值变量上运行,前向与反向计算仅使用位运算。关键优势在于首次实现循环神经网络的端到端二值训练。在多层感知机和循环神经网络上验证,测试准确率分别提升最高达+6.89%和+10.57%。算法已开源。
原文摘要 · Abstract (English)
Binary Neural Networks (BNNs), which constrain both weights and activations to binary values, offer substantial reductions in computational complexity, memory footprint, and energy consumption. These advantages make them particularly well suited for deployment on resource-constrained devices. However, training BNNs via gradient-based optimization remains challenging due to the discrete nature of their variables. The dominant approach, quantization-aware training, circumvents this issue by employing surrogate gradients. Yet, this method requires maintaining latent full-precision parameters and performing the backward pass with floating-point arithmetic, thereby forfeiting the efficiency of binary operations during training. While alternative approaches based on local learning rules exist, they are unsuitable for global credit assignment and for back-propagating errors in multi-layer architectures. This paper introduces Binary Error Propagation (BEP), the first learning algorithm to establish a principled, discrete analog of the backpropagation chain rule. This mechanism enables error signals, represented as binary vectors, to be propagated backward through multiple layers of a neural network. BEP operates entirely on binary variables, with all forward and backward computations performed using only bitwise operations. Crucially, this makes BEP the first solution to enable end-to-end binary training for recurrent neural network architectures. We validate the effectiveness of BEP on both multi-layer perceptrons and recurrent neural networks, demonstrating gains of up to +6.89% and +10.57% in test accuracy, respectively. The proposed algorithm is released as an open-source repository.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。