让手机在1GB内存内高效微调大模型，比零阶优化快10倍以上
Memory-Efficient Backpropagation for Fine-Tuning LLMs on Resource-Constrained Mobile Devices
- 提出MeBP方法，在移动端实现低内存反向传播
- 0.5B~4B参数模型仅需<1GB内存，训练速度比零阶优化快10~100倍
- 适合移动端微调大模型的研究者与开发者
即使仅微调部分参数（如LoRA），使用反向传播微调大语言模型（LLM）的内存消耗仍远高于推理，通常被认为不适合资源受限的移动设备。替代方法如零阶优化（ZO）虽大幅降低内存占用，但收敛速度显著变慢（比反向传播多10~100倍迭代步数）。本文提出一种移动端内存高效的反向传播实现（MeBP），在内存与计算时间间取得更好平衡，收敛更快且性能更优。我们在iPhone 15 Pro Max上验证了其有效性，证明0.5B至4B参数的多种LLM可仅用不到1GB内存完成微调。相关代码已开源：https://github.com/apple/ml-mebp。
原文摘要 · Abstract (English)
Fine-tuning large language models (LLMs) with backpropagation\textemdash even for a subset of parameters such as LoRA\textemdash can be much more memory-consuming than inference and is often deemed impractical for resource-constrained mobile devices. Alternative methods, such as zeroth-order optimization (ZO), can greatly reduce the memory footprint but come at the cost of significantly slower model convergence (10$\times$ to 100$\times$ more steps than backpropagation). We propose a memory-efficient implementation of backpropagation (MeBP) on mobile devices that provides better trade-off between memory usage and compute time, while converging faster and achieving better performance than the ZO baseline. We verify the effectiveness of MeBP on an iPhone 15 Pro Max and show that various LLMs, ranging from 0.5B to 4B parameters, can be fine-tuned using less than 1GB of memory. We release an example of the MeBP implementation at https://github.com/apple/ml-mebp.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。