提出新型自适应优化器，通信量降低170倍且抗系统故障。
DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Models
- 参数与动量分设不同同步周期，减少通信开销。
- 在17亿参数模型上通信量仅为DDP的1/170。
- 无需同步全部优化器状态，适合实际训练场景。
使用分布式数据并行（DDP）方法扩展基础模型训练时，带宽成为瓶颈。现有稀疏通信方法如Local SGD仅同步模型参数，无法直接应用于自适应优化器，因其需额外同步优化器状态。当前扩展Local SGD的方法要么缺乏收敛保证，要么需同步所有优化器状态，通信成本增至三倍。本文提出异步低通信自适应优化器（DES-LOC），为参数和动量分别设定独立同步周期，在保持收敛性的同时显著降低通信开销。在规模达17亿参数的语言模型上，实验表明DES-LOC的通信量比DDP减少170倍，比先前最优的Local ADAM减少2倍。与以往启发式方法不同，DES-LOC适用于易发生系统故障的实际训练环境。该方法提供了可扩展、高带宽效率且具备容错能力的基础模型训练方案。
原文摘要 · Abstract (English)
Scaling foundation model training with Distributed Data Parallel (DDP) methods is bandwidth-limited. Existing infrequent communication methods like Local SGD were designed to synchronize only model parameters and cannot be trivially applied to adaptive optimizers due to additional optimizer states. Current approaches extending Local SGD either lack convergence guarantees or require synchronizing all optimizer states, tripling communication costs. We propose Desynced Low Communication Adaptive Optimizers (DES-LOC), a family of optimizers assigning independent synchronization periods to parameters and momenta, enabling lower communication costs while preserving convergence. Through extensive experiments on language models of up to 1.7B, we show that DES-LOC can communicate 170x less than DDP and 2x less than the previous state-of-the-art Local ADAM. Furthermore, unlike previous heuristic approaches, DES-LOC is suited for practical training scenarios prone to system failures. DES-LOC offers a scalable, bandwidth-efficient, and fault-tolerant solution for foundation model training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。