四项技术让大模型在手机上也能私有微调，内存需求降28倍。
Techniques for Peak Memory Reduction for LoRA Fine-tuning of LLMs on Edge Devices
- 用动态反量化、内存优化检查点等四法压内存
- 实测在30亿参数模型上峰值内存降26至28倍
- 适合想在手机上本地微调大模型的开发者
在终端设备上使用低秩适配（LoRA）微调大语言模型可实现个性化且保护数据隐私，但受限于消费级硬件的严重内存瓶颈。微调过程中的峰值内存常超出设备限制，尤其对数十亿参数模型和长上下文训练数据而言。本文提出一组互补技术，在不损失模型质量的前提下降低内存占用：(1) 基础模型量化并配合实时反量化；(2) 结合选择性激活缓存与磁盘卸载的高效检查点机制；(3) 通过语义相关词子集近似软最大值；(4) 对输出逻辑值进行掩码处理。在 Llama-3.2 3B 和 Qwen-2.5 3B 模型上的实验表明，峰值内存最高可减少26倍和28倍，使资源受限设备上的微调成为可能。
原文摘要 · Abstract (English)
Fine-tuning of Large Language Models (LLMs) using Low-Rank Adaptation (LoRA) on an end-user's data offers personalized experiences while keeping data private, but faces severe memory constraints on consumer hardware. Peak memory during fine-tuning often exceeds device limits, especially for models with billions of parameters and long-context training data. This paper introduces a suite of complementary techniques to reduce memory footprint without sacrificing model quality: (1) base model quantization with on-the-fly dequantization, (2) memory-efficient checkpointing combining selective activation caching and disk offloading, (3) softmax approximation using semantically relevant token subsets, and (4) logits masking. Experiments on Llama-3.2 3B and Qwen-2.5 3B demonstrate up to $26\times$ and $28\times$ reduction in peak memory, enabling fine-tuning on resource-constrained devices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。