提出非阻塞通信机制，提升低带宽下大模型训练效率与稳定性。
Factored Gossip DiLoCo: Reducing Blocking Communication in DiLoCo
- 用近似同步的混合通信替代精确同步，降低阻塞风险。
- 在低带宽下训练千亿参数语言模型，计算利用率显著提升。
- 适合分布式训练中对容错性和计算效率要求高的场景。
为使大规模分布式训练在高带宽数据中心外也具实用性，必须减少阻塞且高负载的同步通信。尽管DiLoCo通信频率较低，其外部同步仍依赖高带宽且易受慢节点和瞬时故障影响。本文通过引入混合/八卦式近似同步，放宽精确同步要求，使系统在延迟和通信失败下可降级运行。该方法将DiLoCo同步分解为非阻塞的混合步骤（与计算重叠且无过时），以及阻塞的混合步骤（强化工作节点间一致性），实现计算利用率与优化稳定性的可调权衡。在低带宽环境下对高达千亿参数的语言模型进行实验，新框架相比DiLoCo显著提升了计算利用率，训练进度从相当到几乎完全匹配，且对故障更具鲁棒性。
原文摘要 · Abstract (English)
To make large-scale distributed training practical outside high-bandwidth datacenters, we must reduce blocking, high-volume synchronization. While DiLoCo communicates infrequently, its outer synchronization remains bandwidth-heavy and brittle to stragglers and transient failures. We relax exact synchronization to approximate synchronization via mixing/gossip, which degrades gracefully under delays and communication failures. This allows us to factorize DiLoCo synchronization into a non-blocking mixing step that overlaps computation with no staleness, and a blocking mixing step that tightens worker agreement, yielding a tunable trade-off between compute utilization and optimization stability. On up to billion-parameter language models in low-bandwidth settings, our framework substantially improves compute utilization compared to DiLoCo, with training progress ranging from comparable to closely matching it, and is more robust to failures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。