让分布式训练更抗网络突发拥塞,动态调整梯度容忍度。
DBLP: Phase-Aware Bounded-Loss Transport for Burst-Resilient Distributed ML Training
- 根据训练阶段动态调整梯度丢包容忍度,结合模型特性优化通信。
- 相比基线平均提速24.4%,峰值达33.9%,微突发下延迟降低5.88倍。
- 适合大规模分布式训练场景,尤其在高网络波动环境下表现突出。
分布式机器学习训练因千亿至万亿参数模型的普及已成为必要。尽管已有工作从应用层提升训练效率,但往往忽视网络层瞬时拥塞导致的严重尾延迟与训练时间波动,损害了系统的服务质量。现有网络优化对所有梯度同等对待,未能融入模型训练的内在容错特性。本文提出动态有界丢包协议(DBLP),一种面向训练阶段、抗突发、硬件无关的传输协议,将模型级容错能力融入梯度通信。通过在训练不同阶段动态调节梯度丢包容忍度,DBLP显著降低整体训练时间,并缓解瞬时高丢包事件(即微突发)下的尾延迟崩溃。相较于当前最优方案,DBLP在保持相近测试准确率的前提下,可容忍更高丢包率,平均缩短端到端训练时间24.4%,最大达33.9%;在微突发期间,单轮通信延迟最高提速5.88倍,有效防止突发引发的尾延迟激增,维持训练性能稳定。
原文摘要 · Abstract (English)
Distributed machine learning (ML) training has become a necessity with the prevalence of billion to trillion-parameter-scale models. While prior work has improved training efficiency from the ML perspective at the application layer, it often fails to address transient congestion events at the network layer that introduce severe tail latency and training-time variability, thereby undermining the quality of service of distributed ML training systems. Existing network optimizations treat all gradients equally and thus fail to integrate sufficient model-training insights into communication protocol design. In this paper, we present Dynamic Bounded-Loss Protocol (DBLP), a burst-resilient, training-phase-aware, and hardware-agnostic transport protocol that incorporates model-level tolerance properties into gradient communication. By dynamically adjusting gradient loss tolerance across training phases, DBLP reduces overall training time and mitigates tail-latency collapse during transient high-loss events (i.e., microbursts). Compared to the current state-of-the-art solution (baseline), DBLP tolerates significantly higher loss while achieving comparable test accuracy, and reduces end-to-end training time by an average of 24.4% and a maximum of 33.9%. At microburst events, DBLP achieves up to 5.88x single-round communication latency speedups over the baseline, preventing burst-induced tail-latency spikes and maintaining stable training performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。