解决异步联邦学习中全局与本地更新冲突问题,提升模型准确率和训练速度。
Orthogonal Calibration for Asynchronous Federated Learning
- 分离维护全局与客户端模型权重,用正交校准减少更新干扰。
- 实验显示准确率提升9.6%,训练速度比同步方法快12倍。
- 适用于高延迟、数据异构场景,适合追求高效异步训练的研究者。
异步联邦学习通过按需整合更新并根据延迟调整影响,缓解了传统同步聚合的低效问题。由于异步性和数据异构性,全局与局部的学习目标存在本质不一致——全局优化轨迹可能与正在进行的本地更新冲突。现有方法简单地将最新全局权重分发给客户端,可能覆盖本地进展,导致模型漂移。本文提出OrthoFL,一种正交校准框架,解耦全局与局部学习进度,并在合并前最小化干扰。客户端与服务器分别维护独立模型权重。收到更新后,服务器通过移动平均聚合到全局权重。对于客户端权重,服务器计算客户端延迟期间累积的全局权重偏移,并去除与接收到的更新方向对齐的部分,使结果参数位于与客户端更新正交的子空间中,最大限度保留全局进展信息。校准后的全局偏移随后合并到客户端权重以继续训练。大量实验表明,OrthoFL在准确率上提升9.6%,相比同步方法实现12倍加速;且在多种延迟模式和异构场景下,持续优于最先进的异步基线。
原文摘要 · Abstract (English)
Asynchronous federated learning mitigates the inefficiency of conventional synchronous aggregation by integrating updates as they arrive and adjusting their influence based on staleness. Due to asynchrony and data heterogeneity, learning objectives at the global and local levels are inherently inconsistent -- global optimization trajectories may conflict with ongoing local updates. Existing asynchronous methods simply distribute the latest global weights to clients, which can overwrite local progress and cause model drift. In this paper, we propose OrthoFL, an orthogonal calibration framework that decouples global and local learning progress and adjusts global shifts to minimize interference before merging them into local models. In OrthoFL, clients and the server maintain separate model weights. Upon receiving an update, the server aggregates it into the global weights via a moving average. For client weights, the server computes the global weight shift accumulated during the client's delay and removes the components aligned with the direction of the received update. The resulting parameters lie in a subspace orthogonal to the client update and preserve the maximal information from the global progress. The calibrated global shift is then merged into the client weights for further training. Extensive experiments show that OrthoFL improves accuracy by 9.6% and achieves a 12$\times$ speedup compared to synchronous methods. Moreover, it consistently outperforms state-of-the-art asynchronous baselines under various delay patterns and heterogeneity scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。