解决联邦学习中低秩适配的异构客户端聚合难题
ILoRA: Federated Learning with Low-Rank Adaptation for Heterogeneous Client Aggregation
- 用正交初始化确保各客户端起始子空间一致
- 通过拼接分解聚合不同秩的参数，减少误差
- 结合自适应优化器缓解非独立同分布数据下的偏差
在客户端异构条件下，基于低秩适配（LoRA）的联邦学习面临三大挑战：(1) 随机初始化导致客户端子空间错位引发不稳定性；(2) 不同秩的LoRA参数平均产生秩不兼容与聚合误差，偏移全局模型；(3) 在非独立同分布（Non-IID）数据下加剧客户端漂移，损害泛化能力。为此，我们提出ILoRA，一个统一框架，包含三项核心创新：基于QR分解的正交初始化，确保所有客户端起始于一致子空间；拼接式QR聚合机制，通过拼接与分解融合异构秩更新，保持维度对齐并保留信息；以及带秩感知控制变量的AdamW优化器，校正局部更新、缓解客户端漂移。理论分析提供收敛保证，视觉与自然语言处理基准上的大量实验表明，ILoRA在准确率和收敛稳定性上均显著优于现有联邦LoRA方法。
原文摘要 · Abstract (English)
Federated Learning with Low-Rank Adaptation (LoRA) faces three critical challenges under client heterogeneity: (1) Initialization-Induced Instability due to random initialization misaligning client subspaces; (2) Rank Incompatibility and Aggregation Error when averaging LoRA parameters of different ranks, which biases the global model; and (3) exacerbated Client Drift under Non-IID Data, impairing generalization. To address these challenges, we propose ILoRA, a unified framework that integrates three core innovations: a QR-based orthonormal initialization to ensure all clients start in a coherent subspace; a Concatenated QR Aggregation mechanism that fuses heterogeneous-rank updates via concatenation and decomposition, preserving information while maintaining dimension alignment; and an AdamW optimizer with rank-aware control variates to correct local updates and mitigate client drift. Supported by theoretical convergence guarantees, extensive experiments on vision and NLP benchmarks demonstrate that ILoRA consistently achieves superior accuracy and convergence stability compared to existing federated LoRA methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。