arXiv:2507.07114cs.DCcs.LG2025-07被引 2

让大模型训练在丢包环境下仍保持精度和收敛,无需改代码

Distributed Training under Packet Loss

  • 通过两阶段机制重建丢失消息的梯度,保证均值无偏
  • 64卡训练下10%丢包时困惑度仅升0.8%
  • 适合在普通或广域网部署大规模模型训练

当前主流语言与视觉模型常跨数千个GPU训练,分布于多个数据中心,但现有框架依赖可靠连接(如InfiniBand或RoCE)。ACK流量与重传导致尾延迟上升,限制可扩展性。使用不可靠连接虽可降低延迟,却可能损害模型精度与收敛性。本文提出一种新型分布式训练框架,可在真实丢包环境下运行,无需修改模型代码或优化器,实现无偏梯度聚合与有界参数漂移。核心思想是双阶段防御:(i) 每个工作节点从实际到达的报文重建一致的梯度估计,确保期望层面正确;(ii) 证明即使经过任意多轮迭代,节点间模型差异仍为O(1),防止异步系统中常见的无界发散。在LLAMA2 7B模型上64个GPU实验验证:随机丢包10%时,困惑度变化不超过0.8%。本工作弥合了通信高效的数据中心协议与现代大模型训练对精度与泛化保障的需求之间的鸿沟,支持在商品化或广域网络上的鲁棒、高吞吐学习。

原文摘要 · Abstract (English)

State-of-the-art language and vision models are routinely trained across thousands of GPUs, often spanning multiple data-centers, yet today's distributed frameworks still assume reliable connections (e.g., InfiniBand or RoCE). The resulting acknowledgment traffic and retransmissions inflate tail latencies and limit scalability. Leveraging unreliable connections will reduce latency but may sacrifice model accuracy and convergence once packets are dropped. A principled, end-to-end solution that preserves accuracy and convergence guarantees under genuine packet loss has previously been missing. We address this critical gap by introducing a novel distributed training framework capable of operating over unreliable connections, offering unbiased gradient aggregation and bounded parameter drift without modifying model code or optimizers. The key insight is a two-stage defense against missing messages: (i) Unbiased gradient aggregation: each worker reconstructs a consistent gradient estimate from whatever packets arrive, guaranteeing expectation-level correctness; and (ii) Bounded-drift parameter broadcasts: we prove the inter-worker model discrepancy remains O(1) even after arbitrarily many iterations, preventing the unbounded divergence typical of asynchronous setups. Analytical bounds are matched by experiments on the LLAMA2 7B model with 64 GPUs: tolerating 10% random packet loss yields at most 0.8% perplexity change. This work bridges the gap between communication-efficient datacenter protocols and the accuracy and generalization guarantees demanded by modern large-model training, enabling robust, high-throughput learning on commodity or wide-area networks.

分布式训练丢包容忍大模型训练通信优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。