提出新方法在手机上高效微调大模型，省内存还保精度。
Memory-Efficient Structured Backpropagation for On-Device LLM Fine-Tuning
- 手动推导利用低秩结构的反向传播，重算中间结果省存储。
- 相比原有方法平均省49%内存，0.5B模型峰值内存从361MB降至136MB。
- 适合资源受限设备做私密化大模型微调，尤其移动端应用。
在设备端微调可实现大语言模型的隐私保护个性化，但移动设备内存极为有限，通常仅6–12GB供所有任务共享。现有方法在精确梯度（高内存，MeBP）与低内存噪声估计（MeZO）间权衡。本文提出内存高效的结构化反向传播（MeSP），通过人工推导利用LoRA的低秩结构进行反向传播。核心思路是：因投影 $h = xA$ 的秩 $r \ll d_{in}$，可在反向传播时低成本重算，无需存储。MeSP在Qwen2.5系列模型（0.5B–3B）上相较MeBP平均减少49%内存，且计算数学等价梯度。分析表明，MeZO的梯度估计与真实梯度相关性极低（余弦相似度≈0.001），解释其收敛缓慢。对Qwen2.5-0.5B，峰值内存从361MB降至136MB，使此前不可行的微调场景成为可能。
原文摘要 · Abstract (English)
On-device fine-tuning enables privacy-preserving personalization of large language models, but mobile devices impose severe memory constraints, typically 6--12GB shared across all workloads. Existing approaches force a trade-off between exact gradients with high memory (MeBP) and low memory with noisy estimates (MeZO). We propose Memory-efficient Structured Backpropagation (MeSP), which bridges this gap by manually deriving backward passes that exploit LoRA's low-rank structure. Our key insight is that the intermediate projection $h = xA$ can be recomputed during backward at minimal cost since rank $r \ll d_{in}$, eliminating the need to store it. MeSP achieves 49\% average memory reduction compared to MeBP on Qwen2.5 models (0.5B--3B) while computing mathematically identical gradients. Our analysis also reveals that MeZO's gradient estimates show near-zero correlation with true gradients (cosine similarity $\approx$0.001), explaining its slow convergence. MeSP reduces peak memory from 361MB to 136MB for Qwen2.5-0.5B, enabling fine-tuning scenarios previously infeasible on memory-constrained devices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。