提出反向动量初始化，缓解联邦学习异构数据下的偏差问题。
Rethinking the initialization of Momentum in Federated Learning with Heterogeneous Data
- 用向前递增的指数权重替代传统动量累积方式
- 在三个基准数据集上验证了方法对异构性的改善效果
- 适合研究联邦学习优化机制或数据异构场景的读者
数据异构性是联邦学习性能的主要挑战。近年来，基于动量的优化技术被证明能有效缓解该问题。在传统方法中，动量随模型更新传输至服务器并聚合，使本地训练以全局动量初始化，受梯度历史引导。然而我们发现，这种传统动量累积方式在联邦学习系统中存在次优性：动量更侧重近期梯度，导致本地训练末期引入更多偏差梯度。本文提出一种新的局部初始化动量估计方法，命名为反向动量联邦学习（RMFL）。核心思想是将梯度权重随时间向前递增地指数衰减，与传统方式相反。RMFL在三个具有不同异构水平的主流基准数据集上进行了评估，结果表明其能有效缓解异构性带来的负面影响。
原文摘要 · Abstract (English)
Data Heterogeneity is a major challenge of Federated Learning performance. Recently, momentum based optimization techniques have beed proved to be effective in mitigating the heterogeneity issue. Along with the model updates, the momentum updates are transmitted to the server side and aggregated. Therefore, the local training initialized with a global momentum is guided by the global history of the gradients. However, we spot a problem in the traditional cumulation of the momentum which is suboptimal in the Federated Learning systems. The momentum used to weight less on the historical gradients and more on the recent gradients. This however, will engage more biased local gradients in the end of the local training. In this work, we propose a new way to calculate the estimated momentum used in local initialization. The proposed method is named as Reversed Momentum Federated Learning (RMFL). The key idea is to assign exponentially decayed weights to the gradients with the time going forward, which is on the contrary to the traditional momentum cumulation. The effectiveness of RMFL is evaluated on three popular benchmark datasets with different heterogeneity levels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。