提出分块洗牌训练法，提升边缘设备模型收敛与泛化性能。
On the Convergence and Stability of Distributed Sub-model Training
- 将全模型分块后由服务器洗牌分发给客户端，本地更新后聚合。
- 理论证明该方法具有可保证的收敛速度，实验验证其优于随机采样。
- 适合资源受限的联邦学习场景，尤其关注模型稳定性与泛化能力。
随着模型规模持续增大，实现设备端本地训练成为联邦学习的关键挑战。一种流行方案是子模型训练：服务器仅向边缘客户端分发随机采样的子模型，客户端仅更新这些小模型。然而，这种随机采样可能导致收敛性能不佳。本文受随机梯度下降（SGD）中洗牌机制成功的启发，提出分布式洗牌子模型训练：预先将全模型划分为多个子模型，服务器在每轮中对这些子模型进行洗牌，并分别发送给客户端；客户端在本地更新后返回更新后的子模型，服务器进行平均聚合。本文建立了该算法的收敛速率。同时通过稳定性分析研究了分布式子模型训练的泛化能力，发现子模型训练可通过增强训练过程的稳定性来提升泛化性能。大量实验验证了理论结果。
原文摘要 · Abstract (English)
As learning models continue to grow in size, enabling on-device local training of these models has emerged as a critical challenge in federated learning. A popular solution is sub-model training, where the server only distributes randomly sampled sub-models to the edge clients, and clients only update these small models. However, those random sampling of sub-models may not give satisfying convergence performance. In this paper, observing the success of SGD with shuffling, we propose a distributed shuffled sub-model training, where the full model is partitioned into several sub-models in advance, and the server shuffles those sub-models, sends each of them to clients at each round, and by the end of local updating period, clients send back the updated sub-models, and server averages them. We establish the convergence rate of this algorithm. We also study the generalization of distributed sub-model training via stability analysis, and find that the sub-model training can improve the generalization via amplifying the stability of training process. The extensive experiments also validate our theoretical findings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。