为联邦学习设计高效梯度压缩方法,显著降低通信开销。
An Efficient Gradient-Aware Error-Bounded Lossy Compressor for Federated Learning
- 利用训练轮次间时序相关性和卷积核结构规律预测梯度,降低残差熵
- 相比现有方法压缩比提升1.53倍,精度损失更小
- 适用于低带宽场景,适合实际联邦学习系统部署
联邦学习(FL)可在不暴露客户端私有数据的前提下实现协作模型训练,但其部署常受客户端与服务器间梯度传输带来的通信开销限制,尤其在系统异构环境下,低带宽客户端成为性能瓶颈。对梯度数据进行有损压缩可缓解此问题,其中误差有界有损压缩(EBLC)因其精细的效用-压缩权衡而备受关注。然而,现有EBLC方法(如SZ)最初针对具有强空间局部性的平滑科学数据设计,依赖洛伦佐和插值等通用预测器降低熵以提高压缩率。而梯度张量平滑性低、空间相关性弱,导致这些预测器效果差,压缩率不佳。为此,我们提出专用于联邦学习梯度数据的新型EBLC框架,在保持模型精度的同时实现高压缩率。核心是创新的预测机制,利用训练轮次间的时序相关性和卷积核内部结构规律减少残差熵。该预测器兼容标准量化器和熵编码器,包含:(1) 基于归一化指数移动平均的跨轮次幅值预测器;(2) 利用梯度振荡特性和核级符号一致性构建的符号预测器。实验表明,该方法相比SZ3最高可实现1.53倍压缩比提升,且精度损失更低。集成至真实世界联邦学习框架APPFL后,在多种受限带宽场景下,端到端通信时间减少76.1%–96.2%,展现出强大的实际部署可扩展性。
原文摘要 · Abstract (English)
Federated learning (FL) enables collaborative model training without exposing clients' private data, but its deployment is often constrained by the communication cost of transmitting gradients between clients and the central server, especially under system heterogeneity where low-bandwidth clients bottleneck overall performance. Lossy compression of gradient data can mitigate this overhead, and error-bounded lossy compression (EBLC) is particularly appealing for its fine-grained utility-compression tradeoff. However, existing EBLC methods (e.g., SZ), originally designed for smooth scientific data with strong spatial locality, rely on generic predictors such as Lorenzo and interpolation for entropy reduction to improve compression ratio. Gradient tensors, in contrast, exhibit low smoothness and weak spatial correlation, rendering these predictors ineffective and leading to poor compression ratios. To address this limitation, we propose an EBLC framework tailored for FL gradient data to achieve high compression ratios while preserving model accuracy. The core of it is an innovative prediction mechanism that exploits temporal correlations across FL training rounds and structural regularities within convolutional kernels to reduce residual entropy. The predictor is compatible with standard quantizers and entropy coders and comprises (1) a cross-round magnitude predictor based on a normalized exponential moving average, and (2) a sign predictor that leverages gradient oscillation and kernel-level sign consistency. Experiments show that this new EBLC yields up to 1.53x higher compression ratios than SZ3 with lower accuracy loss. Integrated into a real-world FL framework, APPFL, it reduces end-to-end communication time by 76.1%-96.2% under various constrained-bandwidth scenarios, demonstrating strong scalability for real-world FL deployments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。