用二值随机算法训练神经网络，能耗降十倍，适合硬件加速。
Energy-Efficient Supervised Learning with a Binary Stochastic Forward-Forward Algorithm
- 用二值随机单元替代实数激活，将矩阵乘转为高效索引操作。
- 在MNIST等数据集上性能接近实数模型，能耗降低约90%。
- 适配新型磁性比特器件，可实现极低功耗的神经网络训练。
降低现代机器学习的能耗已成为迫切需求，因其依赖于规模更大、能耗更高的神经网络。然而，主流训练算法反向传播因串行依赖和需存储前向激活的内存开销，难以适配专用硬件加速器。虽存在替代方案，但其计算瓶颈转为矩阵乘法。本研究提出适用于二值随机单元的前向-前向算法：通过激活二值化将矩阵乘转化为索引操作，可在硬件中高效执行；结合权重共享与不同偏置的随机性，突破二值单元的信息瓶颈。此外，尽管传统硬件中二值采样缓慢昂贵，但可利用新型不稳定磁体构成的p-bit器件实现高速低成本采样。我们在MNIST、Fashion-MNIST和CIFAR-10数据集上评估该算法，结果表明性能接近实数前向-前向模型，但能耗预计降低约一个数量级。
原文摘要 · Abstract (English)
Reducing energy consumption has become a pressing need for modern machine learning, which has achieved many of its most impressive results by scaling to larger and more energy-consumptive neural networks. Unfortunately, the main algorithm for training such networks, backpropagation, poses significant challenges for custom hardware accelerators, due to both its serial dependencies and the memory footprint needed to store forward activations for the backward pass. Alternatives to backprop, although less effective, do exist; here the main computational bottleneck becomes matrix multiplication. In this study, we derive forward-forward algorithms for binary, stochastic units. Binarization of the activations transforms matrix multiplications into indexing operations, which can be executed efficiently in hardware. Stochasticity, combined with tied weights across units with different biases, bypasses the information bottleneck imposed by binary units. Furthermore, although slow and expensive in traditional hardware, binary sampling that is very fast can be implemented cheaply with p-bits (probabilistic bits), novel devices made up of unstable magnets. We evaluate our proposed algorithms on the MNIST, Fashion-MNIST, and CIFAR-10 datasets, showing that its performance is close to real-valued forward-forward, but with an estimated energy savings of about one order of magnitude.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。