让手机等设备高效微调大模型，靠冻结部分模型+服务器低成本更新。
SplitFrozen: Split Learning with Device-side Model Frozen for Fine-Tuning LLM on Heterogeneous Resource-Constrained Devices
- 设备只做前向计算，冻结部分模型层，只上传中间特征。
- 在极端数据不均衡下准确率比现有方法高69.4%，设备计算量降86.8%。
- 适合资源受限的异构设备，尤其适合个性化大模型微调场景。
在私有、本地数据上微调大语言模型（LLMs）可实现个性化的智能代理。然而，在资源受限的边缘设备上微调LLMs面临计算开销大、设备异构性及数据不平衡等挑战。本文提出SplitFrozen，一种分片学习框架，通过在设备端冻结模型层并由服务器集中进行参数高效微调，实现高效微调。该框架将模型分为设备端冻结层与服务器端可微调层，异构资源受限设备仅执行前向传播。为降低服务器训练成本，引入低秩适配（LoRA）于服务器端层；采用流水线并行策略解耦设备-服务器计算，并利用分解反向传播优化训练效率。在GPT-2上使用MRPC、MNLI-matched和SST-2数据集的实验表明，当数据极度不均衡时，SplitFrozen相比FedLoRA和SplitLoRA提升69.4%模型准确率，设备端计算量减少86.8%，总训练时间缩短50.2%。在使用Llama-3.2模型与GSM8K数据集的内容生成任务中，也验证了SplitFrozen的可扩展性。
原文摘要 · Abstract (English)
Fine-tuning large language models (LLMs) on private, on-device data can empower tailored personalized AI agents. However, fine-tuning LLMs on resource-constrained edge devices faces significant challenges, including excessive computation overhead, device heterogeneity, and data imbalance. This paper proposes SplitFrozen, a split learning framework that enables efficient LLM fine-tuning by strategically freezing device-side model layers while centralizing parameter-efficient fine-tuning on the server. Our framework partitions LLMs into device-side frozen layers and server-side fine-tuning layers, where heterogeneous resource-constrained devices execute only forward propagation. To minimize server-side training costs, we integrate Low-Rank Adaptation (LoRA) into the server-side layers. A pipeline parallelism strategy further optimizes training efficiency by decoupling device-server computations and leveraging decomposed backward propagation. Experiments on GPT-2 with the MRPC, MNLI-matched, and SST-2 datasets demonstrate that SplitFrozen outperforms FedLoRA and SplitLoRA by 69.4\% model accuracy under extremely imbalanced data, while reducing up to 86.8\% device-side computations and 50.2\% total training time. Experiments also validate the scalability of SplitFrozen on content generation task using Llama-3.2 model on GSM8K dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。