让大模型在不稳定的分布式设备上高效训练，支持随时加入或离开的节点。
Go With The Flow: Churn-Tolerant Decentralized Training of Large Language Models
- 用新型去中心化流量算法动态优化训练数据传输路径。
- 在10地异构节点、高掉线率下，训练时间缩短45%。
- 适合资源不均、网络不稳的边缘协同训练场景。
受大语言模型兴起及训练民主化需求推动，我们提出GWTF——首个实用的崩溃容错去中心化大模型训练框架。与现有分布式和联邦训练框架不同，GWTF可在自愿贡献资源的异构客户端间高效协同训练大模型。同时，它解决了节点频繁进出（节点掉线）和网络不稳定问题。核心是创新的去中心化流量算法，可动态寻找最优传输路径，以最小延迟最大化微批次训练数量。我们在类GPT和类LLaMa模型上进行了广泛评估，并与先前方法对比。结果表明，在包含10个地理分布区域、高节点掉线率的真实挑战性场景中，GWTF可将训练时间减少最高达45%。
原文摘要 · Abstract (English)
Motivated by the emergence of large language models (LLMs) and the importance of democratizing their training, we propose GWTF, the first crash tolerant practical decentralized training framework for LLMs. Differently from existing distributed and federated training frameworks, GWTF enables the efficient collaborative training of a LLM on heterogeneous clients that volunteer their resources. In addition, GWTF addresses node churn, i.e., clients joining or leaving the system at any time, and network instabilities, i.e., network links becoming unstable or unreliable. The core of GWTF is a novel decentralized flow algorithm that finds the most effective routing that maximizes the number of microbatches trained with the lowest possible delay. We extensively evaluate GWTF on GPT-like and LLaMa-like models and compare it against the prior art. Our results indicate that GWTF reduces the training time by up to 45% in realistic and challenging scenarios that involve heterogeneous client nodes distributed over 10 different geographic locations with a high node churn rate.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。