arXiv:2410.20495cs.DCcs.LG2024-10

通过智能减少通信次数,让边缘设备更快完成模型训练。

When Less is More: Achieving Faster Convergence in Distributed Edge Machine Learning

  • 根据模型性能提升动态决定是否发送更新,减少无效通信。
  • 实测训练时间缩短13.22倍,通信开销降低62.1%。
  • 适合资源有限、设备异构的边缘AI部署场景。

在资源受限的边缘设备上进行分布式机器学习具有巨大应用潜力,但如何在异构环境中实现快速收敛仍是难题。传统框架如批量同步并行和异步随机并行依赖频繁的小规模更新,导致显著的通信开销,阻碍收敛速度。且这些框架通常采用固定数据集大小,忽视设备异构性,易产生处理缓慢的‘拖延节点’,拖慢整体训练进程。本文提出Hermes,一种新型概率性分布式机器学习框架。该框架基于近期测试损失行为动态设定阈值,仅在检测到模型泛化能力显著提升时才传输更新,大幅降低通信开销。同时,Hermes采用动态数据分配策略,优化资源利用,防止拖延节点引发的性能下降。在真实异构资源受限环境中的评估表明,Hermes相比现有最优方法实现更快收敛,训练时间减少13.22倍,通信开销降低62.1%。

原文摘要 · Abstract (English)

Distributed Machine Learning (DML) on resource-constrained edge devices holds immense potential for real-world applications. However, achieving fast convergence in DML in these heterogeneous environments remains a significant challenge. Traditional frameworks like Bulk Synchronous Parallel and Asynchronous Stochastic Parallel rely on frequent, small updates that incur substantial communication overhead and hinder convergence speed. Furthermore, these frameworks often employ static dataset sizes, neglecting the heterogeneity of edge devices and potentially leading to straggler nodes that slow down the entire training process. The straggler nodes, i.e., edge devices that take significantly longer to process their assigned data chunk, hinder the overall training speed. To address these limitations, this paper proposes Hermes, a novel probabilistic framework for efficient DML on edge devices. This framework leverages a dynamic threshold based on recent test loss behavior to identify statistically significant improvements in the model's generalization capability, hence transmitting updates only when major improvements are detected, thereby significantly reducing communication overhead. Additionally, Hermes employs dynamic dataset allocation to optimize resource utilization and prevents performance degradation caused by straggler nodes. Our evaluations on a real-world heterogeneous resource-constrained environment demonstrate that Hermes achieves faster convergence compared to state-of-the-art methods, resulting in a remarkable $13.22$x reduction in training time and a $62.1\%$ decrease in communication overhead.

边缘计算分布式训练通信优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。