优化无线联邦学习调度,提升精度并减少设备参与数。
FedCGD: Collective Gradient Divergence Optimized Scheduling for Wireless Federated Learning
- 从群体梯度发散角度设计调度策略,兼顾设备与样本两级差异。
- 在CIFAR-10上提升准确率4.2%,设备调度量减少41.8%。
- 适用于资源受限场景,尤其适合异构数据下的高效联邦学习。
联邦学习(FL)允许多个设备协同训练模型。在无线网络中,设备数据异构性和带宽有限性持续影响其性能。现有研究多将数据异构性视为单个设备的属性,本文证明:FL收敛速度受设备级和样本级集体梯度发散(CGD)之和的影响。设备级CGD指被调度设备组的整体梯度差异,而非个体差异之和;样本级CGD由采样方差决定,与总采样样本数成反比。针对分类任务,将设备组分布与全局分布间的差异建模为加权地球移动距离(WEMD),提出FedCGD算法,在多项式时间内平衡WEMD与采样方差,最小化多层级CGD。仿真表明,该策略在CIFAR-10上最高提升分类准确率4.2%,同时仅需调度41.8%的设备,并可灵活切换优化目标。
原文摘要 · Abstract (English)
Federated learning (FL) is a promising paradigm for multiple devices to cooperatively train a model. When applied in wireless networks, two issues consistently affect the performance of FL, i.e., data heterogeneity of devices and limited bandwidth. Many papers have investigated device scheduling strategies considering the two issues. However, most of them recognize data heterogeneity as a property of individual devices. In this paper, we prove that the convergence speed of FL is affected by the sum of device-level and sample-level collective gradient divergence (CGD). The device-level CGD refers to the gradient divergence of the scheduled device group, instead of the sum of the individual device divergence. The sample-level CGD is statistically upper bounded by sampling variance, which is inversely proportional to the total number of samples scheduled for local update. To derive a tractable form of the device-level CGD, we further consider a classification problem and transform it into the weighted earth moving distance (WEMD) between the group distribution and the global distribution. Then we propose FedCGD algorithm to minimize the sum of multi-level CGDs by balancing WEMD and sampling variance, within polynomial time. Simulation shows that the proposed strategy increases classification accuracy on the CIFAR-10 dataset by up to 4.2\% while scheduling 41.8\% fewer devices, and flexibly switches between reducing WEMD and reducing sampling variance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。