解决联邦学习中低秩微调的秩坍缩问题，提升异构客户端性能
Preventing Rank Collapse in Federated Low-Rank Adaptation with Client Heterogeneity
- 按秩分块聚合本地更新，缓解异构客户端影响
- 实验表明性能优于主流基线，对秩配置不敏感
- 适合资源或数据差异大的真实联邦场景
联邦低秩适配（FedLoRA）实现了下游任务中基础模型的高效、隐私保护微调。在实际联邦学习中，客户端在系统资源和数据分布上的异质性促使各客户端采用不同低秩秩值。然而，我们发现基于SVD分配的异构FedLoRA存在一种被忽视的现象——秩坍缩：全局更新的能量集中在最小共享秩上，导致性能下降且对秩配置高度敏感。通过理论分析，我们揭示其根源在于：与秩无关的聚合权重与秩相关的客户端贡献不匹配，使高秩更新以几何速率被系统抑制。为此，我们提出raFLoRA，一种秩分区聚合方法，将本地更新分解为秩分区，并根据有效客户端贡献加权聚合。在视觉、语言和推理任务上的大量实验表明，raFLoRA可防止秩坍缩，提升模型性能，并增强对多样化异构配置的鲁棒性，优于强基线FedLoRA。
原文摘要 · Abstract (English)
Federated low-rank adaptation (FedLoRA) has facilitated communication-efficient and privacy-preserving fine-tuning of foundation models for downstream tasks. In practical federated learning scenarios, client heterogeneity in system resources and data distributions motivates the use of heterogeneous LoRA ranks across clients. However, we identify a previously overlooked phenomenon in heterogeneous FedLoRA with SVD-based allocation, termed rank collapse, where the energy of the global update becomes concentrated in the minimum shared rank, resulting in suboptimal performance and high sensitivity to rank configurations. Through theoretical analysis, we reveal the root cause of rank collapse: a mismatch between rank-agnostic aggregation weights and rank-dependent client contributions, which systematically suppresses higher-rank updates at a geometric rate over rounds. Motivated by this insight, we propose raFLoRA, a rank-partitioned aggregation method that decomposes local updates into rank partitions and then aggregates each partition weighted by its effective client contributions. Extensive experiments across vision, language, and reasoning tasks show that raFLoRA prevents rank collapse, improves model performance, and enhances robustness across diverse heterogeneous configurations compared with strong FedLoRA baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。