解决联邦学习中LoRA在数据异构下的性能下降问题
Rethinking LoRA for Data Heterogeneous Federated Learning: Subspace and State Alignment
- 在客户端用子空间优化，服务端通过谱信号提取同步状态
- 在非独立同分布设置下，准确率比现有方法提升1.8%~4.3%
- 适合数据异构的联邦学习场景，尤其适用于大模型微调
低秩适配（LoRA）广泛用于联邦微调，但在非独立同分布（non-IID）设置下，其性能显著低于全参数微调。通过高概率鲁棒性分析，我们发现这一差距源于两个耦合的不匹配：(i) 更新空间不匹配——客户端在低秩子空间优化，而聚合发生在全空间；(ii) 优化器状态不匹配——未同步的自适应状态放大了跨轮次漂移。为此，我们提出FedGaLore，结合客户端的GaLore式梯度子空间优化与服务端基于谱共享信号提取的投影二阶矩状态鲁棒同步机制。在NLU、视觉和自然语言生成（NLG）多个基准上，FedGaLore在非IID设置下均优于现有先进联邦LoRA基线，提升稳健性与精度。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) is widely used for federated fine-tuning. Yet under non-IID settings, it can substantially underperform full-parameter fine-tuning. Through with-high-probability robustness analysis, we uncover that this gap can be attributed to two coupled mismatches: (i) update-space mismatch, where clients optimize in a low-rank subspace but aggregation occurs in the full space; and (ii) optimizer-state mismatch, where unsynchronized adaptive states amplify drift across rounds. We propose FedGaLore, which combines client-side GaLore-style gradient-subspace optimization with server-side drift-robust synchronization of projected second-moment states via spectral shared-signal extraction, to address this challenge. Across NLU, vision, and NLG benchmarks, FedGaLore improves robustness and accuracy over state-of-the-art federated LoRA baselines in non-IID settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。