利用梯度时空相关性,显著降低联邦学习通信开销。
Communication-Efficient Federated Learning by Exploiting Spatio-Temporal Correlations of Gradients
- 通过分解梯度为基向量与系数,压缩数据量
- 仅传输少量更新的基向量和系数,通信量减少39.79%
- 适合带宽受限场景,保持收敛速度与精度
通信开销是联邦学习在带宽受限网络中的关键挑战。现有方法多聚焦于单个梯度的压缩,忽视了相邻轮次间梯度的时序相关性。实证分析发现,梯度具有强时序相关性。基于此,提出GradESTC压缩技术,同时利用梯度的空间与时序相关性:先用空间相关性将全梯度分解为紧凑基向量与组合系数;再利用时序相关性,每轮仅动态更新少量基向量。相比完整梯度传输,该方法仅需发送轻量级系数与少量更新基向量,大幅降低通信量。大量实验表明,在接近收敛目标精度时,相较于最强基线,上行通信平均减少39.79%,且收敛速度与最终精度与未压缩的FedAvg相当。通过有效利用梯度的时空结构,GradESTC为高效联邦学习提供了实用且可扩展的解决方案。
原文摘要 · Abstract (English)
Communication overhead is a critical challenge in federated learning, particularly in bandwidth-constrained networks. Although many methods have been proposed to reduce communication overhead, most focus solely on compressing individual gradients, overlooking the temporal correlations among them. Prior studies have shown that gradients exhibit spatial correlations, typically reflected in low-rank structures. Through empirical analysis, we further observe a strong temporal correlation between client gradients across adjacent rounds. Based on these observations, we propose GradESTC, a compression technique that exploits both spatial and temporal gradient correlations. GradESTC exploits spatial correlations to decompose each full gradient into a compact set of basis vectors and corresponding combination coefficients. By exploiting temporal correlations, only a small portion of the basis vectors need to be dynamically updated in each round. GradESTC significantly reduces communication overhead by transmitting lightweight combination coefficients and a limited number of updated basis vectors instead of the full gradients. Extensive experiments show that, upon reaching a target accuracy level near convergence, GradESTC reduces uplink communication by an average of 39.79% compared to the strongest baseline, while maintaining comparable convergence speed and final accuracy to uncompressed FedAvg. By effectively leveraging spatio-temporal gradient structures, GradESTC offers a practical and scalable solution for communication-efficient federated learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。