arXiv:2509.26399cs.AI2025-09

提出FLoRA-NA,让联邦低秩微调更准更省通信。

Communication-Efficient and Accurate Approach for Aggregation in Federated Low-Rank Adaptation

  • 用服务器端近似聚合矩阵,避免客户端频繁传参
  • 在不增加通信量下提升全局性能,超越现有方法
  • 适合需要高效个性化微调的分布式大模型场景

随着基础模型兴起和分布式微调需求增长,联邦低秩适应(FedLoRA)受到广泛关注。但现有方法存在更新不精确问题,导致局部-全局泛化差距大且通信开销高,制约其可扩展性。为此,我们提出FLoRA-NA:利用服务器端本地LoRA矩阵估计聚合矩阵$\hat{A}$和$\hat{B}$,再分发给客户端进行本地更新。该代理聚合矩阵使实际更新$\nabla \hat{W} = \hat{B}\hat{A}$与理想更新$\nabla \Bar{W} = \sum^{U}_{u=1}B_u A_u$尽可能接近,无需额外通信成本。实验在自然语言理解、数学推理和代码生成等任务上验证了效果,使用多种基础模型,结果一致表明FLoRA-NA在保持低通信开销的同时达到当前最优全局性能。

原文摘要 · Abstract (English)

With the rapid emergence of foundation models and the increasing need for fine-tuning across distributed environments, Federated Low-Rank Adaptation (FedLoRA) has recently gained significant attention. Despite enormous potential, current FedLoRA methods face notable challenges due to inexact updates. Existing approaches have attempted to mitigate this issue, but they often introduce a \emph{local-global generalization gap} and incur \emph{substantial communication overhead}, limiting their scalability and effectiveness. To address these limitations, we propose \textbf{F}ederated \textbf{Lo}w-\textbf{R}ank \textbf{A}ggregation with \textbf{N}early \textbf{A}ccurate Estimation (FLoRA-NA). FLoRA-NA leverages the local LoRA matrices on the server to estimate the aggregated matrices $\hat{A}$ and $\hat{B}$, which are then distributed to clients for local updates. This surrogated aggregated matrices minimizes the divergence between ideal $\nabla \Bar{W} = \sum^{U}_{u=1}B_u A_u$ and practical updates $\nabla \hat{W} = \hat{B}\hat{A}$ without adding communication cost beyond vanilla FedLoRA. By doing so, FLoRA-NA achieves communication efficiency and bridges the gap between local personalization and global generalization, addressing a key limitation of prior personalized FedLoRA approaches. We conduct extensive evaluations across diverse tasks, including natural language understanding, mathematical reasoning, and code-solving ability using various foundation models. Experimental results consistently demonstrate that FLoRA-NA achieves state-of-the-art global performance while maintaining low communication overhead.

联邦学习低秩微调通信效率

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。