解决联邦学习中低秩更新的分布错位问题，提升模型收敛与性能。
Subspace-Constrained Federated Learning with Low-Rank Adaptation
- 通过共享参考子空间约束本地更新方向，缓解数据异构带来的几何错位。
- 在RoBERTa-large上实现0.454的最高准确率，最终损失降至1.363。
- 首次量化验证子空间对齐对联邦学习的关键作用，适合关注隐私训练的研究者。
联邦低秩适配方法在通信与隐私受限下微调大模型极具吸引力，但客户端数据异构会导致局部低秩更新之间的几何错位。本文研究这种子空间错位是否引发有害聚合及收敛变慢。提出一种基于子空间正则化的联邦LoRA目标，强制本地更新贴近共享全局参考子空间。在两个预训练模型（RoBERTa-large 和 SmolLM-360M）上，于非独立同分布的10客户端联邦设置下进行完整实验，覆盖3个随机种子（42, 43, 44），共24次运行（4种方法 × 3种子 × 2模型）。在RoBERTa-large上，Subspace-Reg实现均值最佳准确率0.454±0.023、均值最终准确率0.429±0.011、最低最终损失1.363，显著优于FedAvg、SVD重分配和FedSVD基线。在SmolLM-360M上，FedAvg表现最优，表明增益具有模型依赖性。关键的是，Subspace-Reg在所有设置下达到约0.9999的基底重叠度，远超基线的0.958–0.991，为几何对齐假设提供有力支持。代码已开源。
原文摘要 · Abstract (English)
Federated low-rank adaptation methods are attractive for fine-tuning large models under communication and privacy constraints, but heterogeneous client data can induce geometric misalignment between local low-rank updates. We study whether this subspace misalignment leads to destructive aggregation and slower convergence in LoRA-based federated learning. We propose a subspace-regularized federated LoRA objective that encourages local client updates to remain close to a shared global reference subspace. We present a complete empirical evaluation on two pretrained models, RoBERTa-large and SmolLM-360M, over HellaSwag in a non-IID 10-client federated setting, across 3 random seeds (42, 43, 44), yielding 24 total experimental runs (4 methods x 3 seeds x 2 models). On RoBERTa-large, Subspace-Reg achieves the strongest mean best accuracy (0.454 +/- 0.023), mean final accuracy (0.429 +/- 0.011), and lowest final loss (1.363) across all three seeds, outperforming FedAvg, SVD redistribution, and FedSVD baselines by a large margin. On SmolLM-360M, FedAvg leads on accuracy, revealing that accuracy gains are model-dependent. Crucially, Subspace-Reg achieves near-perfect basis overlap, approximately 0.9999, on both models and across all seeds, versus 0.958 to 0.991 for all baselines, providing robust support for the geometric alignment hypothesis. The code is publicly available at https://github.com/sadia-sigma-lab/Subspace-Constrained-Federated-learning-with-Lora.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。