提出分层循环梯度计算，让手机端微调大模型更快更省内存。
LCSB: Layer-Cyclic Selective Backpropagation for Memory-Efficient On-Device LLM Fine-Tuning
- 每步只计算部分层的梯度，利用残差连接保证信息传递。
- 比全量反向传播快1.4倍，精度损失小于2%。
- 4比特量化下更稳定，有隐式正则化效果，适合移动端部署。
内存高效反向传播（MeBP）已实现仅用不到1GB内存在移动设备上对大语言模型进行一阶微调。然而，MeBP每步需通过所有Transformer层进行反向计算，其中权重解压缩就占了反向时间的32%–42%。我们提出分层循环选择性反向传播（LCSB），每步仅计算部分层的梯度。关键洞察是：残差连接确保梯度通过恒等路径流动，而AdamW动量为未选层提供隐式更新。我们将LCSB解释为在LoRA参数空间上的块坐标下降，提供了收敛性理论依据。LCSB在五个模型和三个任务上实现最高1.40×加速，精度损失低于2%。令人惊讶的是，在4比特量化设置下，LCSB表现出更好稳定性：一个在全量反向传播下完全发散的30亿参数模型，使用LCSB后能平稳收敛，表明选择性梯度计算具有隐式正则化效应。
原文摘要 · Abstract (English)
Memory-efficient backpropagation (MeBP) has enabled first-order fine-tuning of large language models (LLMs) on mobile devices with less than 1GB memory. However, MeBP requires backward computation through all transformer layers at every step, where weight decompression alone accounts for 32--42% of backward time. We propose Layer-Cyclic Selective Backpropagation (LCSB), which computes gradients for only a subset of layers per step. Our key insight is that residual connections guarantee gradient flow through identity paths, while AdamW momentum provides implicit updates for non-selected layers. We interpret LCSB as Block Coordinate Descent on the LoRA parameter space, providing theoretical justification for convergence. LCSB achieves up to 1.40$\times$ speedup with less than 2\% quality degradation across five models and three tasks. Surprisingly, in 4-bit quantized settings, LCSB exhibits superior stability: a 3B model that completely diverges under full backpropagation converges smoothly with LCSB, suggesting an implicit regularization effect from selective gradient computation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。