只更新重要通道，大幅降低边缘设备训练内存占用。
Dynamic Gradient Sparse Update for Edge Training
- 动态选择重要通道与层进行梯度更新，跳过次要部分。
- 仅更新2%卷积权重，仍达85.77%准确率，内存减少98%。
- 适合资源受限的边缘设备，如微控制器上的模型微调。
在边缘设备上训练可实现个性化模型微调，提升实际性能并保护数据隐私。然而，反向传播中的梯度计算需要大量内存来存储中间特征和损失值，这对内存受限的边缘设备（如微控制器）不可接受。为此，我们提出一种基于动态梯度稀疏更新的训练加速方法：仅对重要通道和层进行梯度更新，跳过不重要部分以降低每次迭代的内存消耗。同时，通道选择随迭代动态变化，沿时间维度遍历更多参数以提升性能。实验表明，该方法使预训练的MobileNetV2在CIFAR-10上训练时仅更新2%的卷积权重，在256KB片上内存下达到85.77%准确率，相比密集训练内存使用减少98%。
原文摘要 · Abstract (English)
Training on edge devices enables personalized model fine-tuning to enhance real-world performance and maintain data privacy. However, the gradient computation for backpropagation in the training requires significant memory buffers to store intermediate features and compute losses. This is unacceptable for memory-constrained edge devices such as microcontrollers. To tackle this issue, we propose a training acceleration method using dynamic gradient sparse updates. This method updates the important channels and layers only and skips gradient computation for the less important channels and layers to reduce memory usage for each update iteration. In addition, the channel selection is dynamic for different iterations to traverse most of the parameters in the update layers along the time dimension for better performance. The experimental result shows that the proposed method enables an ImageNet pre-trained MobileNetV2 trained on CIFAR-10 to achieve an accuracy of 85.77\% while updating only 2\% of convolution weights within 256KB on-chip memory. This results in a remarkable 98\% reduction in feature memory usage compared to dense model training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。