提出LoRA-A²方法，解决异构数据下低秩微调的通信与性能难题。
Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clients
- 采用交替冻结与自适应秩选择策略，提升异构环境鲁棒性。
- 低秩条件下仍保持性能，参数上传量比全微调减少显著。
- 适合资源受限场景下的大模型联邦微调应用。
大语言模型（LLMs）的联邦微调面临模型更新传输带来的巨大通信开销。尽管低秩适配（LoRA）被提出作为解决方案，但在联邦学习中因聚合不一致而复杂化。现有方法在异构数据设置下低秩时常导致性能下降。为此，我们提出LoRA-A²（低秩适配中的交替冻结与自适应秩选择），在低秩和高数据异构性条件下表现出强鲁棒性。实验表明，即使在极端异构性和低秩情况下，LoRA-A²仍能保持性能，相比全微调可显著减少上传参数量，且不损失性能。该自适应机制提升了联邦微调的鲁棒性与通信效率，使大模型在资源受限环境中的实际部署成为可能。
原文摘要 · Abstract (English)
Federated fine-tuning for Large Language Models (LLMs) faces significant challenges due to the heavy communication overhead of transmitting large model updates. Although Low Rank Adaptation (LoRA) has been proposed as a solution, yet its application in federated learning is complicated by discordance in aggregation. Existing methods addressing this discordance often suffer from performance degradation at low ranks in heterogeneous data settings. In response, we introduce LoRA-A$^2$ (Low Rank Adaptation with Alternating freeze and Adaptive rank selection), which demonstrates robustness in challenging settings with low ranks and high data heterogeneity. Our experimental findings reveal that LoRA-A$^2$ maintains performance even under extreme heterogeneity and low rank conditions, achieving up to a significant reduction in uploaded parameters compared to full fine-tuning without compromising performance. This adaptive mechanism increases robustness and communication efficiency in federated fine-tuning, enabling the practical deployment of LLMs in resource-constrained environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。