arXiv:2607.14367cs.LG2026-07

解决联邦学习中LoRA适配器干扰问题,提升模型聚合稳定性。

Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learning

论文配图:Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learning
图 1 · 摘自论文原文
  • 动态分配客户端专属低秩子空间,避免参数冲突。
  • 在合成任务上训练损失降低9倍,临床文本分类准确率提升4.3%。
  • 无需修改原有框架,仅增加0.9%运行开销,适合医疗等场景。

联邦微调大模型普遍采用低秩适配(LoRA)以降低通信与计算开销,但客户端异质性会导致适配器聚合不稳定。我们识别出数据-参数干扰是该不稳定的几何根源,其受LoRA更新子空间与客户端激活之间的对齐影响。因此,联邦LoRA聚合不仅是参数平均,更是子空间分配。为此提出动态子空间增强方法(Dysco),可在联邦环境中动态分配客户端特定的LoRA子空间。每轮中,客户端从本地表示中计算出对激活不敏感的子空间,并仅传输基向量;服务器通过闭式解构建客户端专属融合子空间,最大化与其他客户端无关方向的兼容性。为应对表示漂移,Dysco进行多轮子空间增强,保留历史更新方向并适应未来表示。我们提供了收敛性分析,将数据-参数干扰嵌入标准联邦优化边界中的聚合误差项,并证明Dysco的服务器固定融合子空间可得到更紧的误差上界。在可控合成任务和基于Llama-3.2-1B的MIMIC-IV临床笔记分类实验中,Dysco显著减少干扰,在正交子空间划分条件下使最终轮次合成训练损失降低达9倍;在五个测试联邦算法上均提升最多4.3%,优于近期联邦LoRA方法,且仅增加0.9%实际运行开销。代码已开源。

原文摘要 · Abstract (English)

Federated fine-tuning of large pre-trained models increasingly relies on Low-Rank Adaptation (LoRA) to reduce communication and computation, but heterogeneous clients can make adapter aggregation unstable. We identify the data-parameter interference as a geometric source of this instability. This interference is controlled by the alignment between LoRA update subspaces and client activations, suggesting that federated LoRA aggregation should be viewed not only as parameter averaging but also as subspace allocation. We propose Dynamic Subspace Boosting (Dysco), a plug-in method that allocates client-specific LoRA subspaces in a federated and dynamic manner. In each round, clients compute activation-insensitive subspaces from local representations and transmit only the resulting bases; the server then constructs client-specific merged subspaces through a closed-form solution that maximizes compatibility with other clients' insensitive directions. To handle representation drift, Dysco performs multi-round subspace boosting to preserve past update directions while adapting to future representations. We provide a convergence analysis that embeds the data-parameter interference as an aggregation-error term in a standard federated optimization bound, and prove that Dysco's server-fixed merged subspaces yield a tighter upper bound on this error. Experiments on controlled synthetic federated tasks and on MIMIC-IV clinical-note classification with Llama-3.2-1B show that Dysco substantially reduces interference, reduces the final-round synthetic training loss by up to 9 times relative to baselines under the orthogonal-subspace partition the theory identifies, improves all five tested FL algorithms by up to 4.3% on MIMIC, outperforms recent federated LoRA methods, and adds only 0.9% wall-clock overhead. Our code is available at https://github.com/illidanlab/Dysco.

联邦学习LoRA子空间优化医疗AI

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。