通过模型拆分突破异构联邦学习的内存瓶颈,提升移动端训练效率与精度。
Breaking the Memory Wall for Heterogeneous Federated Learning via Model Splitting
- 采用分层架构动态划分模型,按设备内存与算力智能切分训练任务。
- 在真实手机设备上实现94%延迟降低、100倍内存节省,精度提升1.49%~57.18%。
- 适合资源受限的移动端联邦学习场景,尤其适用于内存波动环境。
联邦学习(FL)允许多个设备在保护数据隐私的前提下协同训练共享模型。然而,日益增长的模型复杂度与设备端有限的内存资源严重制约了其在现实场景中的部署。因此,亟需一种能有效突破内存瓶颈,并兼顾硬件与统计异构性的框架。本文提出 SmartSplit,通过模型拆分显著降低设备端内存占用,同时保障训练进度与模型精度。该框架采用分层结构,由服务器端的中央管理器根据各设备的内存预算、训练能力与数据分布动态选择参与设备及切割层;边缘服务器上的 MEC 管理器负责本地模型拆分与服务端部分训练,并基于时变统计重要性优化切点;设备端管理器则持续监控本地训练状态,利用成本感知检查点机制匹配动态内存预算。在代表性数据集和商用手机测试平台上开展的大量实验表明,SmartSplit 在高度内存受限的移动 SoC 上表现优异,峰值延迟降低高达94%,内存消耗减少100倍,精度提升1.49%~57.18%,并通过成本感知重计算自适应调整动态内存预算。
原文摘要 · Abstract (English)
Federated Learning (FL) enables multiple devices to collaboratively train a shared model while preserving data privacy. Ever-increasing model complexity coupled with limited memory resources on the participating devices severely bottlenecks the deployment of FL in real-world scenarios. Thus, a framework that can effectively break the memory wall while jointly taking into account the hardware and statistical heterogeneity in FL is urgently required. In this paper, we propose SmartSplit, a framework that effectively reduces the memory footprint on the device side while guaranteeing the training progress and model accuracy for heterogeneous FL through model splitting.Towards this end, SmartSplit employs a hierarchical structure to adaptively guide the overall training process. In each training round, the central manager, hosted on the server, dynamically selects the participating devices and sets the cutting layer by jointly considering the memory budget, training capacity, and data distribution of each device. The MEC manager, deployed within the edge server, proceeds to split the local model and perform training of the server-side portion. Meanwhile, it fine-tunes the splitting points based on the time-evolving statistical importance. The on-device manager, embedded inside each mobile device, continuously monitors the local training status while employing cost-aware checkpointing to match the runtime dynamic memory budget. Extensive experiments on representative datasets are conducted on both commercial off-the-shelf mobile device testbeds. The experimental results show that SmartSplit excels in FL training on highly memory-constrained mobile SoCs, offering up to a 94% peak latency reduction and 100-fold memory savings. It enhances accuracy performance by 1.49%-57.18% and adaptively adjusts to dynamic memory budgets through cost-aware recomputation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。