用8位整数训练神经网络，让边缘设备更快更省电。
FF-INT8: Efficient Forward-Forward DNN Training on Edge Devices with INT8 Precision
- 用前向-前向算法替代反向传播，减少内存占用。
- 在边缘设备上实现4.6%提速、8.3%省电、27%减存。
- 适合资源受限场景的高效模型训练，如智能终端。
反向传播是神经网络训练的核心方法，但其在时间和能耗上的低效限制了其在资源受限的边缘设备上的应用。尽管低精度量化被广泛研究以加速推理，但在训练中的应用仍较少。最近提出的前向-前向（FF）算法作为反向传播的替代方案，通过增加一次前向传播取代反向传播，避免存储中间激活值，从而降低内存开销，更适合嵌入式设备。本文提出一种基于FF算法的INT8量化训练方法，利用其逐层策略稳定梯度量化。此外，我们设计了一种新颖的“前瞻”机制，解决FF的局限性并提升模型精度。在NVIDIA Jetson Orin Nano板上的实验表明，该方法实现了4.6%的训练加速、8.3%的能效提升以及27.0%的内存减少，同时保持与当前最先进水平相当的准确率。
原文摘要 · Abstract (English)
Backpropagation has been the cornerstone of neural network training for decades, yet its inefficiencies in time and energy consumption limit its suitability for resource-constrained edge devices. While low-precision neural network quantization has been extensively researched to speed up model inference, its application in training has been less explored. Recently, the Forward-Forward (FF) algorithm has emerged as a promising alternative to backpropagation, replacing the backward pass with an additional forward pass. By avoiding the need to store intermediate activations for backpropagation, FF can reduce memory footprint, making it well-suited for embedded devices. This paper presents an INT8 quantized training approach that leverages FF's layer-by-layer strategy to stabilize gradient quantization. Furthermore, we propose a novel "look-ahead" scheme to address limitations of FF and improve model accuracy. Experiments conducted on NVIDIA Jetson Orin Nano board demonstrate 4.6% faster training, 8.3% energy savings, and 27.0% reduction in memory usage, while maintaining competitive accuracy compared to the state-of-the-art.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。