720亿参数模型通过去中心化网络训练，实现无需审批的全球协作。
Covenant-72B: Pre-Training a 72B LLM with Trustless Peers Over-the-Internet
- 用高效通信优化器支持动态加入退出的分布式训练
- 在约1.1万亿词上预训练，性能媲美集中式模型
- 首个开放权限的全球分布式大模型训练，适合研究去中心化AI
近年来，全球分布式训练受到关注，有望降低训练成本并促进大规模基础模型的民主化参与。然而，现有分布式训练模型规模较小，且仅限白名单参与者，未能真正实现开放参与。本文介绍Covenant-72B，这是迄今为止规模最大、计算资源最多的全球分布式预训练运行，首次实现了开放、无许可的参与者协作，并由实时区块链协议支持。我们采用先进的通信高效优化器SparseLoCo，支持动态参与，允许节点自由加入与退出。模型在约1.1万亿文本标记上进行预训练，其性能与同等或更高算力预算的全集中式模型相当，证明了完全去中心化、非白名单化的参与不仅可行，且可达到前所未有的规模。
原文摘要 · Abstract (English)
Recently, there has been increased interest in globally distributed training, which has the promise to both reduce training costs and democratize participation in building large-scale foundation models. However, existing models trained in a globally distributed manner are relatively small in scale and have only been trained with whitelisted participants. Therefore, they do not yet realize the full promise of democratized participation. In this report, we describe Covenant-72B, an LLM produced by the largest collaborative globally distributed pre-training run (in terms of both compute and model scale), which simultaneously allowed open, permissionless participation supported by a live blockchain protocol. We utilized a state-of-the-art communication-efficient optimizer, SparseLoCo, supporting dynamic participation with peers joining and leaving freely. Our model, pre-trained on approximately 1.1T tokens, performs competitively with fully centralized models pre-trained on similar or higher compute budgets, demonstrating that fully democratized, non-whitelisted participation is not only feasible, but can be achieved at unprecedented scale for a globally distributed pre-training run.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。