提出新初始化方法缓解联邦学习中的客户端漂移问题。
Rethinking the Personalized Relaxed Initialization in the Federated Learning: Consistency and Generalization
- 通过反向移动初始化，实现每轮本地训练的个性化松弛初始化。
- 实验证明该方法在不增加通信成本下达到与先进方法相当性能。
- 揭示了局部不一致主要影响泛化误差而非优化误差，适合关注泛化的研究者。
联邦学习（FL）是一种分布式范式，通过多阶段本地训练在异构数据集上协同训练全局模型。以往工作隐含指出FL存在“客户端漂移”问题，由各客户端最优解不一致引起，但缺乏坚实的理论分析。为此，本文提出高效算法FedInit，允许在每轮本地训练开始时采用个性化松弛初始化：将本地状态从当前全局状态向最新本地状态的反方向移动。为进一步理解不一致对性能的影响，引入超出风险分析，研究发散项以探究测试误差。研究表明，优化误差对局部不一致不敏感，而主要影响泛化误差界。大量实验验证了其有效性：所提方法在无需额外训练或通信开销的情况下，性能可媲美多个先进基准。此外，该阶段式个性化松弛初始化可集成至现有先进算法中，进一步提升联邦学习的泛化性能。
原文摘要 · Abstract (English)
Federated learning (FL) is a distributed paradigm that coordinates massive local clients to collaboratively train a global model via stage-wise local training processes on the heterogeneous dataset. Previous works have implicitly studied that FL suffers from the ``client-drift'' problem, which is caused by the inconsistent optimum across local clients. However, till now it still lacks solid theoretical analysis to explain the impact of this local inconsistency. To alleviate the negative impact of ``client drift'' and explore its substance in FL, in this paper, we first propose an efficient FL algorithm FedInit, which allows employing the personalized relaxed initialization state at the beginning of each local training stage. Specifically, FedInit initializes the local state by moving away from the current global state towards the reverse direction of the latest local state. Moreover, to further understand how inconsistency disrupts performance in FL, we introduce the excess risk analysis and study the divergence term to investigate the test error in FL. Our studies show that optimization error is not sensitive to this local inconsistency, while it mainly affects the generalization error bound. Extensive experiments are conducted to validate its efficiency. The proposed FedInit method could achieve comparable results compared to several advanced benchmarks without any additional training or communication costs. Meanwhile, the stage-wise personalized relaxed initialization could also be incorporated into several current advanced algorithms to achieve higher generalization performance in the FL paradigm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。