提出只共享通用知识的低秩适配方法，提升联邦学习效率
Selective Aggregation for Low-Rank Adaptation in Federated Learning
- 仅上传通用知识矩阵A，减少通信开销
- 在多个任务上实现优于基线的性能提升
- 适用于各类低秩适配变体，具有通用性
我们通过分析低秩适配（LoRA）中学习到的A和B矩阵的不对称性，发现矩阵A负责学习通用知识，而矩阵B则捕捉客户端特有知识。基于此，我们提出联邦共享低秩适配（FedSA-LoRA），仅将矩阵A上传至服务器进行聚合，保留本地B矩阵以适应客户端差异。进一步分析发现，该模式在rsLoRA和VeRA等其他LoRA变体中同样成立，因此将方法扩展至FedSA-rsLoRA和FedSA-VeRA。在自然语言理解与生成任务上的大量实验表明，所提方法有效提升模型性能并降低通信成本。代码已开源。
原文摘要 · Abstract (English)
We investigate LoRA in federated learning through the lens of the asymmetry analysis of the learned $A$ and $B$ matrices. In doing so, we uncover that $A$ matrices are responsible for learning general knowledge, while $B$ matrices focus on capturing client-specific knowledge. Based on this finding, we introduce Federated Share-A Low-Rank Adaptation (FedSA-LoRA), which employs two low-rank trainable matrices $A$ and $B$ to model the weight update, but only $A$ matrices are shared with the server for aggregation. Moreover, we delve into the relationship between the learned $A$ and $B$ matrices in other LoRA variants, such as rsLoRA and VeRA, revealing a consistent pattern. Consequently, we extend our FedSA-LoRA method to these LoRA variants, resulting in FedSA-rsLoRA and FedSA-VeRA. In this way, we establish a general paradigm for integrating LoRA with FL, offering guidance for future work on subsequent LoRA variants combined with FL. Extensive experimental results on natural language understanding and generation tasks demonstrate the effectiveness of the proposed method. Our code is available at https://github.com/Pengxin-Guo/FedSA-LoRA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。