FedOptima提升联邦学习资源利用率,减少设备与服务器闲置。
FedOptima: Optimizing Resource Utilization in Federated Learning
- 异步聚合+辅助网络,消除设备间延迟和任务依赖
- 服务器调度均衡贡献,训练加速1.9至21.8倍,闲置降低超90%
- 适合大规模异构设备参与的高效联邦学习场景
联邦学习(FL)系统在服务器和多个设备间进行分布式机器学习,但普遍存在设备与服务器资源利用率低的问题,限制了实际应用。这种低效主要源于两类空闲时间:(i) 服务器与设备间的任务依赖,(ii) 异构设备中的慢响应者(stragglers)。本文提出FedOptima,一个资源优化的联邦学习系统,同时减少这两类空闲时间。其核心是将神经网络部分层的训练从设备卸载到服务器,通过三项创新实现:(1) 设备采用异步聚合独立运行,消除慢响应影响;利用辅助网络实现设备与服务器解耦,减少任务依赖造成的空闲;(2) 服务器通过任务调度器确保所有设备均衡贡献,提升模型精度;(3) 服务器端高效内存管理,提升可扩展性。在图像分类和情感分析任务上,使用CNN和Transformer模型,在多个实验室测试平台上的实验表明:相比四种先进离线与异步联邦学习基线,FedOptima (i) 达到更高或相当准确率,(ii) 训练速度提升1.9倍至21.8倍,(iii) 服务器和设备空闲时间分别减少最高达93.9%和81.8%,(iv) 吞吐量提高1.1倍至2.0倍。
原文摘要 · Abstract (English)
Federated learning (FL) systems facilitate distributed machine learning across a server and multiple devices. However, FL systems have low resource utilization on servers and devices, limiting their practical use in the real world. This inefficiency primarily arises from two types of idle time: (i) task dependency between the server and devices, and (ii) stragglers among heterogeneous devices. This paper introduces FedOptima, a resource-optimized FL system designed to simultaneously minimize both types of idle time; existing systems do not eliminate or reduce both at the same time. FedOptima offloads the training of certain layers of a neural network from a device to a server using three innovations. First, devices operate independently of each other using asynchronous aggregation to eliminate straggler effects, and independently of the server by utilizing auxiliary networks to minimize idle time caused by task dependency. Second, the server performs centralized training using a task scheduler that ensures balanced contributions from all devices, improving model accuracy. Third, an efficient memory management mechanism on the server increases the scalability of the number of participating devices. Extensive experiments are conducted on multiple lab-based testbeds, evaluated on image classification and sentiment analysis tasks with CNNs and Transformers. Compared to four state-of-the-art offloading-based and asynchronous FL baselines, FedOptima (i) achieves higher or comparable accuracy, (ii) accelerates training by 1.9x to 21.8x, (iii) reduces server and device idle time by up to 93.9% and 81.8%, respectively, and (iv) increases throughput by 1.1x to 2.0x.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。