低维子空间优化提升异构数据下的联邦学习效率
Subspace Optimization for Efficient Federated Learning under Heterogeneous Data
- 在低维子空间中进行异构性修正优化，仅用投影量更新
- 理论收敛速率达$ ilde{ ext{O}}(1/T + 1/\ \sqrt{NKT})$，实测精度效率双优
- 适合资源受限场景，尤其适用于大模型异构数据训练
联邦学习在大模型场景下面临通信、内存与计算资源稀缺问题。非独立同分布（non-IID）的客户端数据常引发训练漂移，影响稳定性和性能。现有方法如SCAFFOLD虽能缓解异构性，但带来显著额外通信与内存开销。本文提出子空间优化方法（SSF），在低维子空间中仅使用投影量执行异构性修正优化，同时通过回填式更新保留残差分量，以维持全维度控制信息。在标准光滑性和有界方差假设下，SSF达到非渐近收敛率$ ilde{ ext{O}}(1/T + 1/ \sqrt{NKT})$。实验表明，在异构数据下，该方法实现优异的准确率-效率权衡。
原文摘要 · Abstract (English)
Federated learning increasingly operates in a large-model regime where communication, memory, and computation are all scarce. Typically, non-IID client data induce drift that degrades the stability and performance of local training. Existing remedies such as SCAFFOLD introduce heterogeneity-correction mechanisms to address this challenge, but they incur substantial extra communication and memory overhead. This paper proposes a subspace optimization method for federated learning (SSF), which performs heterogeneity-corrected optimization in a low-dimensional subspace using only projected quantities, while preserving full-dimensional control information through a backfill-style update that retains residual components whenever the active subspace changes. Under standard smoothness and bounded-variance assumptions, SSF attains a non-asymptotic rate of order $\widetilde{\mathcal{O}}(1/T+1/\sqrt{NKT})$. Experiments show favorable accuracy--efficiency trade-offs under heterogeneous data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。