无需反向传播，在微型设备上实现高效低内存训练
Poor Man's Training on MCUs: A Memory-Efficient Quantized Back-Propagation-Free Approach
- 用零阶方法估算量化模型梯度，避开反向传播
- 在1024KB或256KB内存的MCU上达到接近传统训练效果
- 适合内存紧张、追求快速部署的边缘场景
反向传播（BP）是神经网络训练中计算梯度的标准方法。但在FPGA、微控制器（MCUs）和模拟计算平台等边缘设备上实现基于BP的训练面临硬件资源不足、开发周期长以及低精度设置下误差显著等问题。本文提出一种适用于MCU的简单无反向传播训练方案，使边缘训练硬件设计如同推理硬件设计般简便。采用量化零阶方法估计量化模型参数的梯度，克服了低精度BP方案中直通估计器带来的误差。进一步引入节点扰动、稀疏训练等降维技术以提升零阶训练的收敛性。实验表明，该方法在资源受限的边缘设备（如拥有1024-KB SRAM的MCU进行全模型密集训练，或256-KB SRAM进行稀疏训练）上，对预训练图像分类器适应各类噪声数据的表现与基于BP的方法相当。该方法最适用于内存开销和上市时间优先、可容忍较长延迟的应用场景。
原文摘要 · Abstract (English)
Back propagation (BP) is the default solution for gradient computation in neural network training. However, implementing BP-based training on various edge devices such as FPGA, microcontrollers (MCUs), and analog computing platforms face multiple major challenges, such as the lack of hardware resources, long time-to-market, and dramatic errors in a low-precision setting. This paper presents a simple BP-free training scheme on an MCU, which makes edge training hardware design as easy as inference hardware design. We adopt a quantized zeroth-order method to estimate the gradients of quantized model parameters, which can overcome the error of a straight-through estimator in a low-precision BP scheme. We further employ a few dimension reduction methods (e.g., node perturbation, sparse training) to improve the convergence of zeroth-order training. Experiment results show that our BP-free training achieves comparable performance as BP-based training on adapting a pre-trained image classifier to various corrupted data on resource-constrained edge devices (e.g., an MCU with 1024-KB SRAM for dense full-model training, or an MCU with 256-KB SRAM for sparse training). This method is most suitable for application scenarios where memory cost and time-to-market are the major concerns, but longer latency can be tolerated.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。