针对嵌入式设备无浮点运算能力,提出仅用整数训练的新方法。
PRIOT: Pruning-Based Integer-Only Transfer Learning for Embedded Systems
- 通过剪枝而非更新权重优化网络,支持静态量化缩放因子。
- 在旋转MNIST和CIFAR-10上提升准确率8.08至33.75个百分点。
- 适合资源受限的微控制器,如Raspberry Pi Pico,兼顾性能与内存效率。
设备端迁移学习对于将通用骨干模型适配到各边缘设备的独特环境至关重要。微型微控制器(如Raspberry Pi Pico)是设备端学习的关键目标,但通常缺乏浮点单元,需采用纯整数训练。以往研究中动态计算量化缩放因子带来高计算开销。因此,本研究聚焦于使用静态缩放因子的纯整数训练,这在现有方法下具有挑战性。我们提出一种名为PRIOT的新训练方法,通过剪枝选定边而非更新权重来优化网络,实现静态缩放因子下的有效训练。剪枝模式由边弹出算法决定,该算法训练分配给每条边的得分参数,而非原始参数,并在推理前剪除低分边。此外,我们引入一种内存高效的变体PRIOT-S,仅对少量边分配得分。我们在Raspberry Pi Pico上实现了PRIOT和PRIOT-S,使用小型CNN模型在旋转MNIST数据集和VGG11模型在旋转CIFAR-10数据集上评估其准确率与计算成本。结果表明,PRIOT相比现有方法准确率提升8.08至33.75个百分点,而PRIOT-S在极小准确率损失下显著降低内存占用。
原文摘要 · Abstract (English)
On-device transfer learning is crucial for adapting a common backbone model to the unique environment of each edge device. Tiny microcontrollers, such as the Raspberry Pi Pico, are key targets for on-device learning but often lack floating-point units, necessitating integer-only training. Dynamic computation of quantization scale factors, which is adopted in former studies, incurs high computational costs. Therefore, this study focuses on integer-only training with static scale factors, which is challenging with existing training methods. We propose a new training method named PRIOT, which optimizes the network by pruning selected edges rather than updating weights, allowing effective training with static scale factors. The pruning pattern is determined by the edge-popup algorithm, which trains a parameter named score assigned to each edge instead of the original parameters and prunes the edges with low scores before inference. Additionally, we introduce a memory-efficient variant, PRIOT-S, which only assigns scores to a small fraction of edges. We implement PRIOT and PRIOT-S on the Raspberry Pi Pico and evaluate their accuracy and computational costs using a tiny CNN model on the rotated MNIST dataset and the VGG11 model on the rotated CIFAR-10 dataset. Our results demonstrate that PRIOT improves accuracy by 8.08 to 33.75 percentage points over existing methods, while PRIOT-S reduces memory footprint with minimal accuracy loss.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。