arXiv:2605.29460cs.CV2026-05

提出FedSmoothLoRA,让联邦LoRA训练更平滑更快收敛。

FedSmoothLoRA: Toward Smoother and Faster Convergence in Federated Low-Rank Adaptation

论文配图:FedSmoothLoRA: Toward Smoother and Faster Convergence in Federated Low-Rank Adaptation
图 1 · 摘自论文原文
  • 用匹配轮次和梯度对齐矩阵初始化LoRA,保持训练连续性。
  • 在图像分类与文本生成任务上显著提升收敛速度与性能。
  • 适合数据异构、通信受限的联邦学习场景使用。

联邦微调大模型时采用低秩适配(LoRA)可有效降低通信与计算开销并保持数据本地性。然而,直接结合FedAvg与LoRA存在三大问题:更新空间有限,限制模型学习能力;跨轮次状态不匹配,破坏本地优化连续性;客户端无关的初始状态,减缓本地收敛。尽管近期方法通过将LoRA更新融合到主干网络缓解了更新空间限制,但跨轮次状态不匹配与客户端无关起始状态仍未充分解决。为此,本文提出FedSmoothLoRA,一种联邦LoRA微调框架,保留扩展的更新空间,提升跨轮次本地优化连续性,并为本地训练提供客户端感知的起始状态。每轮通信中,该框架利用两个矩阵构建本地LoRA初始化:轮次匹配矩阵以维持跨轮次状态连续性,梯度对齐矩阵则基于本地数据估计的梯度信号提供客户端特异性优化引导。两者协同实现更平滑、更快的收敛。大量实验在图像分类与自然语言生成任务上表明,FedSmoothLoRA始终优于现有联邦LoRA微调方法。

原文摘要 · Abstract (English)

Federated fine-tuning of foundation models with Low-Rank Adaptation (LoRA) provides an efficient solution for reducing communication and computation costs while preserving data locality. However, the direct combination of FedAvg and LoRA suffers from three key issues: limited update space, which restricts the model's effective learning capacity; inter-round state mismatch, which disrupts cross-round local optimization continuity; and a client-agnostic starting state, which slows local convergence on clients. Although recent methods mitigate the limited update space issue by merging LoRA updates into the backbone across communication rounds, inter-round state mismatch and the client-agnostic starting state remain insufficiently addressed. To address these issues, we propose FedSmoothLoRA, a federated LoRA tuning framework that preserves the enlarged update space, improves cross-round local optimization continuity, and provides a client-aware starting state for local training. At each communication round, FedSmoothLoRA constructs the local LoRA initialization using two matrices: a Round-Matching matrix that preserves cross-round local state continuity, and a Gradient-Aligned matrix that provides client-specific optimization guidance from gradient signals estimated on local data. Together, these designs enable smoother and faster convergence. Extensive experiments on image classification and natural language generation tasks demonstrate that FedSmoothLoRA consistently outperforms existing federated LoRA tuning methods. Code: https://github.com/wangzehao0704/FedSmoothLoRA

联邦学习LoRA模型微调高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。