arXiv:2608.15381cs.AIcs.LG2026-08

解决联邦LoRA中聚合与初始化的矛盾,提升异构数据下的模型性能。

FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA

论文配图:FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA
图 1 · 摘自论文原文
  • 通过产品对齐机制,保持本地参数连续性并促进全局一致性。
  • 在异构客户端下平均GLUE得分提升6.82个百分点。
  • 适用于计算和通信资源受限的个性化联邦学习场景。

低秩适配(LoRA)实现了大语言模型的高效联邦微调,但其分解参数化导致本地更新聚合与因子连续性之间的权衡。按因子聚合虽能更好保持因子连续性,但造成聚合不匹配;而产品空间重构可减少该不匹配,却因新重构因子带来更大的初始化不匹配。我们提出FedPA-LoRA,一种产品对齐的联邦LoRA框架,在同质与异质客户端秩条件下均能保证收敛。各客户端在通信轮次中保留本地因子,并将其产品对齐至特定秩的全局参考,既维持本地优化连续性,又在数据异构下促进全局一致性。服务器在公共产品空间中聚合异质秩更新,并高效重建秩约束的全局适配器,无需形成稠密聚合。该设计支持客户端个性化的计算与通信预算。在自然语言理解与生成任务上的实验表明,FedPA-LoRA在不同数据异构程度及同质/异质秩设置下均持续优于代表性基线,异构客户端秩下平均GLUE准确率最高提升6.82个百分点。

原文摘要 · Abstract (English)

Low-Rank Adaptation (LoRA) enables efficient federated fine-tuning of large language models, but its factorized parameterization creates a tension between accurate aggregation of local updates and continuity of locally optimized factors. Factor-wise aggregation incurs aggregation mismatch but better preserves factor continuity, whereas product-space reconstruction reduces this mismatch at the cost of greater factor-level initialization mismatch from newly reconstructed factors. We propose FedPA-LoRA, a product-aligned federated LoRA framework that jointly addresses these limitations and provably converges under both homogeneous and heterogeneous client ranks. Each client preserves its local factors across communication rounds and aligns its product toward a rank-specific global reference, maintaining local optimization continuity while promoting global consistency under data heterogeneity. The server aggregates heterogeneous-rank updates in the common product space and efficiently reconstructs a rank-constrained global adapter without forming the dense aggregate. This design supports client-specific computation and communication budgets. Experiments on natural language understanding and generation tasks show that FedPA-LoRA consistently outperforms representative baselines across varying levels of data heterogeneity and homogeneous- and heterogeneous-rank settings, with up to a $6.82$ percentage-point improvement in average GLUE accuracy under heterogeneous client ranks.

联邦学习LoRA异构数据参数对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。