提出新方法，显著降低设备端训练的内存和计算开销。
Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device Learning
- 用捷径结构替代低秩分解，缓解反向传播激活内存瓶颈。
- 实测激活内存减少最高达120.09倍，训练浮点运算量降低1.86倍。
- 适合资源受限的移动端模型更新，如手机、IoT设备等场景。
设备端学习已成为人工智能发展的重要方向，因其能有效降低延迟、缓解隐私风险并提升能效。然而，内存与计算资源受限仍是其部署的主要障碍。基于此前针对反向传播中激活内存瓶颈的低秩分解研究，本文提出一种新型捷径方法作为替代方案。分析与实验表明，该方法在传统基准测试上可将激活内存使用量降低至原始训练的120.09倍以下，同时训练总浮点运算量（FLOPs）最多减少1.86倍，显著提升设备端学习的可行性。
原文摘要 · Abstract (English)
On-device learning has emerged as a promising direction for AI development, particularly because of its potential to reduce latency issues and mitigate privacy risks associated with device-server communication, while improving energy efficiency. Despite these advantages, significant memory and computational constraints still represent major challenges for its deployment. Drawing on previous studies on low-rank decomposition methods that address activation memory bottlenecks in backpropagation, we propose a novel shortcut approach as an alternative. Our analysis and experiments demonstrate that our method can reduce activation memory usage, even up to $120.09\times$ compared to vanilla training, while also reducing overall training FLOPs up to $1.86\times$ when evaluated on traditional benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。