arXiv:2601.12917cs.LGcs.DC2026-01

让手机端用小模型快速个性化大模型,隐私不泄露。

CooperLLM: Cloud-Edge-End Cooperative Federated Fine-tuning for LLMs via ZOO-based Gradient Correction

  • 手机用零阶优化轻量更新,云端用真实梯度修正偏差。
  • 内存减少86.4%,训练速度提升8.8倍,准确率最高提高10个百分点。
  • 适合资源受限设备做私密化大模型微调,如手机、IoT终端。

大语言模型在众多自然语言任务中表现优异,但因其高内存与计算开销,难以在资源受限的移动设备上进行微调,而用户对隐私保护的个性化需求日益增长。联邦学习可实现本地数据训练,但现有方法或依赖内存密集型反向传播,或采用零阶优化(ZOO),后者虽避免反向传播但收敛慢、精度差。本文提出CooperLLM,一种云-边-端协同的联邦微调框架:移动端使用轻量级ZOO算法在私有数据上更新,云端则在辅助公开数据上通过反向传播训练,并注入引导扰动以校正本地更新,从而提升收敛速度与精度,同时保障隐私。为克服系统瓶颈,CooperLLM引入流水线调度与自适应压缩技术,实现计算与通信重叠,降低内存占用。在多个Transformer模型和数据集上的实验表明,CooperLLM将设备端内存消耗降低高达86.4%,收敛速度提升8.8倍,准确率相比当前最优的ZOO基线最高提升10个百分点。

原文摘要 · Abstract (English)

Large Language Models (LLMs) perform well on many NLP tasks, but fine-tuning them on resource-constrained mobile devices is challenging due to high memory and computation costs, despite growing demands for privacy-preserving personalization. Federated Learning (FL) enables local-data training, yet existing methods either rely on memory-intensive backpropagation or use zeroth-order optimization (ZOO), which avoids backward passes but suffers from slow convergence and degraded accuracy. We propose CooperLLM, a cloud-assisted edge-end cooperative federated fine-tuning framework that combines ZOO on mobile devices with cloud-guided gradient rectification. Mobile clients perform lightweight ZOO updates on private data, while the cloud fine-tunes on auxiliary public data using backpropagation and injects guided perturbations to rectify local updates, improving convergence and accuracy without violating privacy. To address system bottlenecks, CooperLLM introduces pipeline scheduling and adaptive compression to overlap computation and communication and reduce memory usage. Experiments on multiple Transformer models and datasets show that CooperLLM reduces on-device memory by up to $86.4\%$, accelerates convergence by $8.8 \times$, and improves accuracy by up to 10 percentage points over state-of-the-art ZOO-based baselines.

联邦学习大模型微调移动端推理隐私保护

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。