通过推测反向传播加速神经网络训练，提升速度且保持精度
Exploring Parallelism in FPGA-Based Accelerators for Machine Learning Applications
- 在CPU上用OpenMP并行执行前向与推测反向传播
- 阈值0.25时训练时间最快提速24%，单步速度最高快35%
- 适合关注硬件加速与训练效率优化的研究者
推测反向传播作为一种有前景的技术，可通过重叠前向与反向传播步骤来加速神经网络训练。当误差梯度落在特定阈值内时，采用推测权重更新可显著减少训练时间，同时几乎不损失精度。本文在MNIST数据集上使用OpenMP实现推测反向传播，利用其多线程能力并行执行前向与推测反向传播步骤，大幅提高训练速度。该应用计划在先进FPGA上进行综合以验证其硬件加速潜力。基于CPU的实验结果表明，当阈值设为0.25时，执行时间最多提升24%，且在不同轮次下精度仅下降3-4%；与基线相比，单步执行时间最大提速达35%，证明了前向与反向传播重叠的有效性。
原文摘要 · Abstract (English)
Speculative backpropagation has emerged as a promising technique to accelerate the training of neural networks by overlapping the forward and backward passes. Leveraging speculative weight updates when error gradients fall within a specific threshold reduces training time without substantially compromising accuracy. In this work, we implement speculative backpropagation on the MNIST dataset using OpenMP as the parallel programming platform. OpenMP's multi-threading capabilities enable simultaneous execution of forward and speculative backpropagation steps, significantly improving training speed. The application is planned for synthesis on a state-of-the-art FPGA to demonstrate its potential for hardware acceleration. Our CPU-based experimental results demonstrate that speculative backpropagation achieves a maximum speedup of 24% in execution time when using a threshold of 0.25, and accuracy remaining within 3-4% of the baseline across various epochs. Additionally, when comparing individual step execution time, speculative backpropagation yields a maximum speedup of 35% over the baseline, demonstrating the effectiveness of overlapping forward and backward passes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。