揭示联邦学习中模型稳定与优化的权衡关系,指导算法设计提升泛化能力。
Understanding Generalization of Federated Learning: the Trade-off between Model Stability and Optimization
- 提出新分析框架Libra,研究模型稳定性与梯度范数的动态关系。
- 发现增大本地步长或动量虽加速收敛,但降低模型稳定性,反而提升过拟合风险。
- 适用于关注联邦学习泛化性能、超参数调优的研究者与工程师。
联邦学习(FL)是一种分布式学习方法,在保护本地数据隐私的同时跨多个设备训练模型。然而,由于数据异构性,客户端常出现不一致的局部最优,导致收敛行为异常和泛化性能下降。现有研究多通过梯度范数的收敛分析或算法稳定性来解释该问题,但难以准确刻画非凸神经网络训练下联邦学习的泛化表现。为此,本文提出创新的通用化动态分析框架Libra,用于算法依赖的过拟合风险最小化,揭示模型稳定性与梯度范数之间的权衡关系。在标准联邦优化框架及其带服务器动量的变体上,我们证明:更大的本地步长或动量可加速梯度范数收敛,却恶化模型稳定性,从而带来更优的过拟合风险。在标准联邦设置下的实验验证了理论洞见,为超参数调优与未来算法设计提供指导。
原文摘要 · Abstract (English)
Federated Learning (FL) is a distributed learning approach that trains machine learning models across multiple devices while keeping their local data private. However, FL often faces challenges due to data heterogeneity, leading to inconsistent local optima among clients. These inconsistencies can cause unfavorable convergence behavior and generalization performance degradation. Existing studies often describe this issue through \textit{convergence analysis} on gradient norms, focusing on how well a model fits training data, or through \textit{algorithmic stability}, which examines the generalization gap. However, neither approach precisely captures the generalization performance of FL algorithms, especially for non-convex neural network training. In response, this paper introduces an innovative generalization dynamics analysis framework, namely \textit{Libra}, for algorithm-dependent excess risk minimization, highlighting the trade-offs between model stability and gradient norms. We present Libra towards a standard federated optimization framework and its variants using server momentum. Through this framework, we show that larger local steps or momentum accelerate convergence of gradient norms, while worsening model stability, yielding better excess risk. Experimental results on standard FL settings prove the insights of our theories. These insights can guide hyperparameter tuning and future algorithm design to achieve stronger generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。