提出自适应选择共享因子的联邦LoRA方法,提升模型微调效果
Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approach

- 根据输入/输出空间需求动态选择共享A或B矩阵
- 在多种数据分布下性能优于固定共享策略
- 通过秩感知指标提前判断最优共享方案,适合异构联邦学习
低秩适配(LoRA)通过两个紧凑矩阵因子$A$和$B$表示大语言模型的更新,为联邦学习中的高效微调提供可能。受LoRA因子不对称角色启发,本文研究应共享$A$而保留$B$(Share-A/Local-B),还是共享$B$而保留$A$(Share-B/Local-A)。基于最小二乘近似,发现Share-A/Local-B要求客户端更新矩阵共用输入侧秩-$r$空间,而Share-B/Local-A要求输出侧秩-$r$空间。两者投影残差不同,优选残差总和更小的策略。据此提出联邦自适应因子共享低秩适配(FedAS-LoRA),训练前选择共享侧以提升微调性能。设计秩感知共享子空间充分性(RSS)指标,利用冻结LLM主干提取的表示评估共享秩-$r$输入空间是否足够。跨任务、数据分布、LoRA秩与参与设置的实验验证了RSS有效性及FedAS-LoRA优越性。
原文摘要 · Abstract (English)
Low-rank adaptation (LoRA) represents large language model (LLM) updates with two compact matrix factors, i.e., $A$ and $B$, providing an efficient way to fine-tune large models in federated learning paradigm. Inspired by the asymmetric roles of the LoRA factors, we study whether $A$ should be shared across clients while $B$ remains client-specific (Share-A/Local-B), or whether $B$ should instead be shared while $A$ remains client-specific (Share-B/Local-A). With a least-squares surrogate, we reveal that Share-A/Local-B requires the client-specific LoRA update matrices to use a common rank-$r$ input-side space, whereas Share-B/Local-A requires a common rank-$r$ output-side space. The two strategies therefore incur different projection residuals, indicating that the preferred strategy is the one with the smaller aggregate residual across clients. With this insight, we propose Federated Adaptive Factor Sharing Low-Rank Adaptation (FedAS-LoRA), which selects the sharing side before training to enhance fine-tuning performance. To enable adaptive factor selection before training, we design a Rank-Aware Shared-Subspace Sufficiency (RSS) metric, which effectively assesses whether a shared rank-$r$ input subspace is sufficient for the local data distributions using representations extracted from a frozen LLM backbone. Experiments across different tasks, data distributions, LoRA ranks, and participation settings confirm the effectiveness of RSS and the superior performance of FedAS-LoRA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。