让边缘设备几秒内完成模型微调，精度不降。
Skip2-LoRA: A Lightweight On-device DNN Fine-tuning Method for Low-cost Edge Devices
- 在层间插入低秩适配器，动态跳过重复计算
- 微调时间减少90%，在15美元单板上仅需数秒
- 适合资源受限的嵌入式设备部署
本文提出Skip2-LoRA，一种轻量级深度神经网络微调方法，旨在缩小预训练模型与部署模型之间的差距。该方法在最后一层与每一层之间插入可训练的低秩适配器（LoRA），增强网络表达能力的同时保持反向传播开销低。该架构能缓存前向传播的中间结果，随着训练轮次推进，可跳过已见过样本的前向计算。我们将该架构与缓存机制结合，实现Skip2-LoRA，并在一台15美元的单板计算机上进行测试。结果表明，相比参数量相同的基线方法，Skip2-LoRA平均将微调时间缩短90.0%，且保持相同精度，整个过程仅耗时数秒。
原文摘要 · Abstract (English)
This paper proposes Skip2-LoRA as a lightweight fine-tuning method for deep neural networks to address the gap between pre-trained and deployed models. In our approach, trainable LoRA (low-rank adaptation) adapters are inserted between the last layer and every other layer to enhance the network expressive power while keeping the backward computation cost low. This architecture is well-suited to cache intermediate computation results of the forward pass and then can skip the forward computation of seen samples as training epochs progress. We implemented the combination of the proposed architecture and cache, denoted as Skip2-LoRA, and tested it on a $15 single board computer. Our results show that Skip2-LoRA reduces the fine-tuning time by 90.0% on average compared to the counterpart that has the same number of trainable parameters while preserving the accuracy, while taking only a few seconds on the microcontroller board.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。