用交替优化提升LoRA在联邦微调中的鲁棒性，降低通信开销。
Robust Federated Finetuning of Foundation Models via Alternating Minimization of LoRA
- 通过交替最小化优化LoRA，仅更新少量参数。
- 在参数减少和数据异构下仍保持高精度与稳定性。
- 适合资源受限的分布式场景，尤其适用于隐私敏感应用。
参数高效微调（PEFT）作为一种创新训练策略，仅更新部分模型参数，显著降低计算和内存开销，并减少联邦学习中因更新量大带来的通信负担。本文研究了现有将知名PEFT方法LoRA与联邦微调结合时的局限性，提出RoLoRA——一种基于交替最小化的鲁棒联邦微调框架，有效应对微调参数减少和数据异构性增加的挑战。实验表明，RoLoRA不仅具备通信优势，还在多种联邦微调场景中显著提升了鲁棒性与有效性。
原文摘要 · Abstract (English)
Parameter-Efficient Fine-Tuning (PEFT) has risen as an innovative training strategy that updates only a select few model parameters, significantly lowering both computational and memory demands. PEFT also helps to decrease data transfer in federated learning settings, where communication depends on the size of updates. In this work, we explore the constraints of previous studies that integrate a well-known PEFT method named LoRA with federated fine-tuning, then introduce RoLoRA, a robust federated fine-tuning framework that utilizes an alternating minimization approach for LoRA, providing greater robustness against decreasing fine-tuning parameters and increasing data heterogeneity. Our results indicate that RoLoRA not only presents the communication benefits but also substantially enhances the robustness and effectiveness in multiple federated fine-tuning scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。