通过动态压缩降低联邦学习通信开销,保持模型精度
Caesar: A Low-deviation Compression Approach for Efficient Federated Learning
- 按设备模型陈旧度自适应调整下载压缩比
- 根据本地数据量和标签分布决定梯度上传压缩率
- 减少设备等待时间,适合移动端与边缘设备部署
压缩是缓解联邦学习通信开销的有效手段。然而现有方法在压缩过程中因信息损失导致模型/梯度偏差,显著降低训练性能,尤其在数据异构和模型过时情况下更为严重。为此,我们提出Caesar框架,采用低偏差压缩策略。针对全局模型下载,设计贪心算法根据本地模型陈旧度优化各设备的压缩比,确保本地训练初始模型精度;针对本地梯度上传,利用设备本地数据属性(样本数量与标签分布)量化梯度重要性,指导压缩比设定。此外,结合细粒度批处理大小优化,显著减少同步屏障下的设备空等时间。我们在40部智能手机和80个NVIDIA Jetson设备上实现并验证了Caesar。实验表明,相比基于压缩的基线方法,Caesar可在保持相同目标精度的前提下,降低25.54%~37.88%的通信流量,且最终测试精度仅下降0.68%。
原文摘要 · Abstract (English)
Compression is an efficient way to relieve the tremendous communication overhead of federated learning (FL) systems. However, for the existing works, the information loss under compression will lead to unexpected model/gradient deviation for the FL training, significantly degrading the training performance, especially under the challenges of data heterogeneity and model obsolescence. To strike a delicate trade-off between model accuracy and traffic cost, we propose Caesar, a novel FL framework with a low-deviation compression approach. For the global model download, we design a greedy method to optimize the compression ratio for each device based on the staleness of the local model, ensuring a precise initial model for local training. Regarding the local gradient upload, we utilize the device's local data properties (\ie, sample volume and label distribution) to quantify its local gradient's importance, which then guides the determination of the gradient compression ratio. Besides, with the fine-grained batch size optimization, Caesar can significantly diminish the devices' idle waiting time under the synchronized barrier. We have implemented Caesar on two physical platforms with 40 smartphones and 80 NVIDIA Jetson devices. Extensive results show that Caesar can reduce the traffic costs by about 25.54%$\thicksim$37.88% compared to the compression-based baselines with the same target accuracy, while incurring only a 0.68% degradation in final test accuracy relative to the full-precision communication.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。