FedHybrid通过混合技术降低联邦学习内存占用,提升移动端训练效率。
FedHybrid: Breaking the Memory Wall of Federated Learning via Hybrid Tensor Management
- 根据设备内存、算力和数据多样性选择参与方,动态生成执行计划。
- 结合重计算与压缩技术,内存受限下模型准确率提升39.1%,耗时减少15.5倍。
- 适合资源受限的移动设备部署,尤其适用于内存紧张的联邦学习场景。
联邦学习(FL)作为一种新型学习范式,使多个设备在保护数据隐私的前提下协同训练共享模型。然而,内存限制是阻碍其在移动设备上部署的核心挑战。本文提出FedHybrid框架,在不牺牲模型精度和训练进度的前提下,有效降低训练过程中的内存占用。具体而言,FedHybrid首先综合评估设备的内存预算、计算能力和数据多样性,筛选每轮参与的设备;随后分析计算图,为每个选定客户端生成执行计划,结合重计算与压缩技术,根据张量特性动态优化内存使用并最小化训练延迟。本地训练过程中,采用精心设计的激活压缩技术,实现内存高效降低且精度损失最小。我们在模拟环境和真实移动设备上进行了广泛实验。结果表明,相较于基线方法,FedHybrid在不同内存预算下,模型准确率最高提升39.1%,实际运行时间减少15.5倍。
原文摘要 · Abstract (English)
Federated Learning (FL) emerges as a new learning paradigm that enables multiple devices to collaboratively train a shared model while preserving data privacy. However, one fundamental and prevailing challenge that hinders the deployment of FL on mobile devices is the memory limitation. This paper proposes \textit{FedHybrid}, a novel framework that effectively reduces the memory footprint during the training process while guaranteeing the model accuracy and the overall training progress. Specifically, \textit{FedHybrid} first selects the participating devices for each training round by jointly evaluating their memory budget, computing capability, and data diversity. After that, it judiciously analyzes the computational graph and generates an execution plan for each selected client in order to meet the corresponding memory budget while minimizing the training delay through employing a hybrid of recomputation and compression techniques according to the characteristic of each tensor. During the local training process, \textit{FedHybrid} carries out the execution plan with a well-designed activation compression technique to effectively achieve memory reduction with minimum accuracy loss. We conduct extensive experiments to evaluate \textit{FedHybrid} on both simulation and off-the-shelf mobile devices. The experiment results demonstrate that \textit{FedHybrid} achieves up to a 39.1\% increase in model accuracy and a 15.5$\times$ reduction in wall clock time under various memory budgets compared with the baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。