提出异步优化方法，突破分布式训练通信瓶颈。
AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism
- 异步更新数据与流水线并行，降低设备共置要求
- 10亿参数模型上性能接近同步基线，通信开销大幅下降
- 适合大规模模型训练，尤其在低速网络环境下
数据并行与流水线并行是扩展神经网络训练的关键策略，但其高通信成本依赖高速互联的共置计算集群，限制了可扩展性。本文通过在两种并行维度上引入异步更新，放松了共置要求，代价是引入流水线阶段与数据并行副本间的延迟。为缓解延迟问题，针对流水线并行采用权重前瞻机制，针对数据并行提出基于指数移动平均校正的异步稀疏平均方法。本文为稀疏平均和异步更新提供了收敛性保证。在高达10亿参数的大规模语言模型上的实验表明，该方法性能接近完全同步基线，同时显著降低通信开销。
原文摘要 · Abstract (English)
Data and pipeline parallelism are key strategies for scaling neural network training across distributed devices, but their high communication cost necessitates co-located computing clusters with fast interconnects, limiting their scalability. We address this communication bottleneck by introducing asynchronous updates across both parallelism axes, relaxing the co-location requirement at the expense of introducing staleness between pipeline stages and data parallel replicas. To mitigate staleness, for pipeline parallelism, we adopt a weight look-ahead approach, and for data parallelism, we introduce an asynchronous sparse averaging method equipped with an exponential moving average based correction mechanism. We provide convergence guarantees for both sparse averaging and asynchronous updates. Experiments on large-scale language models (up to \em 1B parameters) demonstrate that our approach matches the performance of the fully synchronous baseline, while significantly reducing communication overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。