arXiv:2508.01348cs.LGcs.AI2025-08被引 2

分析LoRA联邦学习中参数聚合的收敛性,揭示不同方法优劣。

Convergence Analysis of Aggregation-Broadcast in LoRA-enabled Distributed Fine-Tuning

  • 提出统一框架分析LoRA联邦学习的聚合策略
  • 证明SP和PS方法分别满足弱/强收敛条件
  • 实验验证理论结论,适合研究联邦学习优化者

联邦学习(FL)可在保护数据隐私的前提下实现跨分散数据源的协同模型训练。然而,机器学习模型规模增长带来了通信与计算挑战。低秩适配(LoRA)作为一种高效的微调方法被引入联邦学习,通过仅更新少量可训练参数来降低通信开销。尽管有效,如何在服务器端聚合本地更新的LoRA模型仍是关键且研究不足的问题。本文为基于LoRA的联邦学习提供统一收敛性分析:首先将现有聚合方法分为两类——求和-乘积(SP)与乘积-求和(PS);随后正式定义聚合-广播算子(ABO),在温和假设下推导出弱收敛与强收敛条件。进一步,分别给出保证本地模型与全局模型收敛的弱、强收敛条件。理论分析揭示了不同聚合策略的本质差异。值得注意的是,我们证明SP与PS方法分别满足弱收敛与强收敛条件,但在达到最优收敛速率的能力上存在差异。在标准基准上的大量实验验证了理论发现。

原文摘要 · Abstract (English)

Federated Learning (FL) enables collaborative model training across decentralized data sources while preserving data privacy. However, the growing size of Machine Learning (ML) models poses communication and computation challenges in FL. Low-Rank Adaptation (LoRA) has recently been introduced into FL as an efficient fine-tuning method, reducing communication overhead by updating only a small number of trainable parameters. Despite its effectiveness, how to aggregate LoRA-updated local models on the server remains a critical and understudied problem. In this paper, we provide a unified convergence analysis for LoRA-based FL. We first categories the current aggregation method into two major type: Sum-Product (SP) and Product-Sum (PS). Then we formally define the Aggregation-Broadcast Operator (ABO) and derive both weak and strong convergence condition under mild assumptions. Furthermore, we present both weak and strong convergence condition that guarantee convergence of the local model and the global model respectively. These theoretical analyze offer a principled understanding of various aggregation strategies. Notably, we prove that the SP and PS aggregation methods satisfy the weak and strong convergence condition respectively, but differ in their ability to achieve the optimal convergence rate. Extensive experiments on standard benchmarks validate our theoretical findings.

联邦学习LoRA收敛分析模型聚合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。