针对多模型联邦学习中的异构客户端采样难题,提出高效低开销的优化方法。
Towards Optimal Heterogeneous Client Sampling in Multi-Model Federated Learning
- 基于损失设计采样策略,兼顾训练方差与通信计算约束。
- 实验显示平均准确率提升最高达19.1%,接近理论最优(仅差5.4%)。
- 适合资源受限的边缘设备部署,尤其适用于多模型协同训练场景。
联邦学习(FL)使边缘设备可在不共享本地数据的情况下协作训练模型。随着FL广泛应用,客户端可能需同时训练多个无关模型,但通信限制使其难以并行训练所有模型。若客户端按需顺序训练模型,可实现多模型联邦学习(MMFL),从而减少整体训练时间。以往工作采用简单的客户端-模型分配方式,未优化各客户端对各模型训练过程的贡献。已有单模型FL研究表明智能客户端选择可显著加速收敛,但直接扩展至MMFL会违反服务器与客户端的异构资源约束。本文首次对任意客户端采样方法下的MMFL进行收敛性分析,揭示了经典梯度方法的优劣。基于此,提出MMFL-LVR:一种基于损失的采样方法,在最小化训练方差的同时显式满足服务器通信限制,并降低客户端计算成本。进一步提出MMFL-StaleVR,引入滞后更新以提升效率与稳定性;以及轻量级变体MMFL-StaleVRE,适用于低开销部署。实验表明,相比随机采样,所提方法平均准确率提升最高达19.1%,与理论最优(全客户端参与)仅相差5.4%。
原文摘要 · Abstract (English)
Federated learning (FL) allows edge devices to collaboratively train models without sharing local data. As FL gains popularity, clients may need to train multiple unrelated FL models, but communication constraints limit their ability to train all models simultaneously. While clients could train FL models sequentially, opportunistically having FL clients concurrently train different models -- termed multi-model federated learning (MMFL) -- can reduce the overall training time. Prior work uses simple client-to-model assignments that do not optimize the contribution of each client to each model over the course of its training. Prior work on single-model FL shows that intelligent client selection can greatly accelerate convergence, but naïve extensions to MMFL can violate heterogeneous resource constraints at both the server and the clients. In this work, we develop a novel convergence analysis of MMFL with arbitrary client sampling methods, theoretically demonstrating the strengths and limitations of previous well-established gradient-based methods. Motivated by this analysis, we propose MMFL-LVR, a loss-based sampling method that minimizes training variance while explicitly respecting communication limits at the server and reducing computational costs at the clients. We extend this to MMFL-StaleVR, which incorporates stale updates for improved efficiency and stability, and MMFL-StaleVRE, a lightweight variant suitable for low-overhead deployment. Experiments show our methods improve average accuracy by up to 19.1% over random sampling, with only a 5.4% gap from the theoretical optimum (full client participation).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。