让低资源设备也能高效微调大模型，兼顾隐私与性能。
FedSODA: Federated Fine-tuning of LLMs via Similarity Group Pruning and Orchestrated Distillation Alignment
- 通过相似性分组剪枝保留关键层，减少模型冗余。
- 通信开销降低70.6%，存储减少75.6%，准确率提升3.1%。
- 适合边缘设备上私密数据的大模型微调场景。
联邦微调（FFT）大语言模型（LLM）在保护数据隐私的同时实现领域适配，但受限于客户端资源，全模型微调的高算力和内存需求限制了其应用。本文提出FedSODA，一种资源高效的FFT框架，使客户端无需访问或存储完整模型即可完成适配。首先提出相似性分组剪枝（SGP）模块，剪除冗余层并保留关键层以维持性能；其次引入协同蒸馏对齐（ODA）模块，缓解子模型与全模型间梯度偏差。结合QLoRA技术，客户端仅需部署量化后的子模型并微调轻量适配器，显著降低本地资源消耗。在三个开源大模型上开展的大量实验表明，FedSODA平均降低70.6%通信开销，减少75.6%存储使用，并将任务准确率提升3.1%，适用于资源受限环境下的实际联邦微调应用。
原文摘要 · Abstract (English)
Federated fine-tuning (FFT) of large language models (LLMs) has recently emerged as a promising solution to enable domain-specific adaptation while preserving data privacy. Despite its benefits, FFT on resource-constrained clients relies on the high computational and memory demands of full-model fine-tuning, which limits the potential advancement. This paper presents FedSODA, a resource-efficient FFT framework that enables clients to adapt LLMs without accessing or storing the full model. Specifically, we first propose a similarity group pruning (SGP) module, which prunes redundant layers from the full LLM while retaining the most critical layers to preserve the model performance. Moreover, we introduce an orchestrated distillation alignment (ODA) module to reduce gradient divergence between the sub-LLM and the full LLM during FFT. Through the use of the QLoRA, clients only need to deploy quantized sub-LLMs and fine-tune lightweight adapters, significantly reducing local resource requirements. We conduct extensive experiments on three open-source LLMs across a variety of downstream tasks. The experimental results demonstrate that FedSODA reduces communication overhead by an average of 70.6%, decreases storage usage by 75.6%, and improves task accuracy by 3.1%, making it highly suitable for practical FFT applications under resource constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。