Ampere让设备端模型训练更快更省通信,还能提升非独立同分布数据下的准确率。
Ampere: Communication-Efficient and High-Accuracy Split Federated Learning
- 采用单向块间训练,用局部损失替代梯度传递,减少通信次数。
- 在非独立同分布数据下准确率最高提升11.7个百分点,通信开销降低911倍。
- 适合资源受限设备部署,尤其适用于数据异构性强的场景。
联邦学习(FL)系统可在设备与服务器间协同训练神经网络,但面临显著的设备端计算负担。分割式联邦学习(SFL)通过将部分网络层卸载至服务器缓解此问题,却因频繁交换中间激活值和梯度带来巨大通信开销,并在非独立同分布(non-IID)数据下降低模型精度。本文提出Ampere,一种新型协同训练系统,同时降低设备端计算与设备-服务器通信开销,并提升模型准确率。不同于SFL采用全局损失进行迭代端到端训练,Ampere采用单向块间训练,分别以局部损失顺序训练设备与服务器端模型块,消除梯度传输。其轻量级辅助网络生成方法解耦设备与服务器训练过程,将频繁的中间激活交换压缩为单次传输,显著降低通信开销。Ampere通过整合已训练设备端生成的激活值来训练服务器端模型,克服SFL在非独立同分布数据上依赖设备特定激活的问题。在多个CNN与Transformer上的实验表明:相比现有最优SFL基线,Ampere(i)准确率最高提升11.70百分点,训练速度加快18.6倍;(ii)设备-服务器通信开销降低最多达911倍,设备端计算减少14.5倍;(iii)不同非独立同分布程度下准确率标准差降低71.13%,展现出更强的异构数据鲁棒性。代码开源:https://github.com/blessonvar/Ampere。
原文摘要 · Abstract (English)
A Federated Learning (FL) system collaboratively trains neural networks across devices and a server but is limited by significant on-device computation costs. Split Federated Learning (SFL) systems mitigate this by offloading a block of layers of the network from the device to a server. However, in doing so, it introduces large communication overheads due to frequent exchanges of intermediate activations and gradients between devices and the server and reduces model accuracy for non-IID data. We propose Ampere, a novel collaborative training system that simultaneously minimizes on-device computation and device-server communication while improving model accuracy. Unlike SFL, which uses a global loss by iterative end-to-end training, Ampere develops unidirectional inter-block training to sequentially train the device and server blocks with a local loss, eliminating the transfer of gradients. A lightweight auxiliary network generation method decouples training between the device and server, reducing frequent intermediate exchanges to a single transfer, which significantly reduces the communication overhead. Ampere mitigates the impact of data heterogeneity by consolidating activations generated by the trained device block to train the server block, in contrast to SFL, which trains on device-specific, non-IID activations. Extensive experiments on multiple CNNs and Transformers show that, compared to state-of-the-art SFL baseline systems, Ampere (i) improves model accuracy by up to 11.70 percentage points while training up to 18.6x faster, (ii) incurs up to 911x lower device-server communication overhead and up to 14.5x lower on-device computation, and (iii) reduces standard deviation of accuracy by 71.13% for various non-IID degrees highlighting superior performance when faced with heterogeneous data. Ampere is available from https://github.com/blessonvar/Ampere.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。