arXiv:2606.24722cs.AI2026-06

用分块训练让小团队也能高效搞大模型,不靠中心化算力

Decentralised AI Training and Inference with BlockTrain

  • 模型拆成可独立训练的块,本地优化后合成完整模型
  • 在WikiText上达到1.359交叉熵,仅比端到端模型低0.04
  • 适合资源有限的研究组或分布式部署场景

前沿人工智能训练正越来越依赖密集且由中心控制的加速器集群,这使超大规模企业与大型实验室占据结构性优势,而开放或独立的AI研究则受限于稀缺资本、特权基础设施和数据中心地理分布。我们提出Spheroid BlockTrain,一种去中心化训练协议:将模型分割为可独立训练的模块,每个模块基于相同全局目标进行本地优化,并在推理时组合成完整模型。在字节级WikiText数据集上,BlockTrain实现1.359的交叉熵(困惑度3.89),距离同配置端到端Transformer参考模型仅差约0.04个交叉熵单位;六名工作者并行训练时,通过聚合同块更新得到1.385交叉熵。基于HTTP/TCP传输的真实序列化检查点与更新实验表明,一次包含三个公网主机的公开IP运行,将交叉熵从5.580降至1.811,共传输15.22 GB数据。推理阶段,当前方案每生成完整输出执行一次块栈遍历,可在三台公网GPU主机间直接通过TCP服务高达758亿参数的逻辑fp16模型,性能优于匹配的纯自回归流水线基线,因其每次广域网遍历可输出一整序列而非单个词元。

原文摘要 · Abstract (English)

Frontier AI training is increasingly shaped by access to dense, centrally controlled accelerator clusters. This creates a structural advantage for hyperscalers and large centralized laboratories, and makes open or independent AI efforts depend on scarce capital, privileged infrastructure, and data-center geography. We present Spheroid BlockTrain, a decentralized training protocol in which a model is partitioned into independently trainable blocks, each optimized on a local objective derived from the same global target and composed at inference into one model. On byte-level WikiText, BlockTrain reaches cross entropy 1.359 (perplexity 3.89), within about 0.04 CE of a same-setup end-to-end Transformer reference, while each active worker trains only one block and avoids full-model optimizer state. A shared six-worker block training run reaches CE 1.385 by averaging same-block updates into one assembled model. HTTP/TCP transport experiments move real serialized checkpoints and updates, including a public-IP three-host run that improves CE from 5.580 to 1.811 while moving 15.22 GB. For inference, the current BlockTrain path uses one block-stack traversal per full output and serves over direct TCP across three public-network GPU hosts up to a 75.80B-parameter logical fp16 shape, outperforming a matched plain-autoregressive TCP pipeline baseline because it emits a full sequence per WAN pipeline traversal rather than one token per traversal.

去中心化训练分块模型分布式推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。