用激励机制让分布式矿工协作训练大模型，突破单机显存限制。
Incentivised Orchestrated Training Architecture (IOTA): A Technical Primer for Release
- 将矿工分层并流式传输激活值，实现模型规模随参与人数无限扩展。
- 激活值压缩最高达128倍，结合蝴蝶形全归约提升训练速度与可扩展性。
- 按贡献精细分配奖励，防止模型囤积，适合去中心化训练研究者参考。
2024年8月，Bittensor的Subnet 9（SN9）证明了基于区块链的去中心化预训练是可行的，其分布式网络中的无许可参与者各自训练了参数量在7亿至140亿之间的大语言模型，并超越了现有基线。但该方案存在两大核心问题：(i) 每个矿工需本地存储完整模型；(ii) “赢家通吃”奖励机制导致模型囤积。本文提出IOTA（Incentivized Orchestrated Training Architecture），通过将原本孤立的矿工整合为协同工作的统一单元，解决了上述限制，实现任意规模扩展并公平激励。关键成果包括：(1) 数据与流水线并行的SWARM架构——协调器将模型层分发至异构矿工，并流式传输激活值，使模型规模可随参与节点数增长，不再受限于单机显存；(2) 细粒度连续激励——验证者量化各矿工贡献，按比例分配代币发行；(3) 激活值压缩——利用模型瓶颈将激活通信带宽降低至原值的1/128，显著提升训练效率；(4) Butterfly All-Reduce——矿工以O(1)带宽完成不重叠参数片的平均，实现线性可扩展性、冗余与内置合谋检测；(5) CLASP（路径采样贡献损失评估）——一种公平归因机制，根据边际效用分配信用，即使贡献相互依赖仍能识别异常行为。
原文摘要 · Abstract (English)
In August 2024, Bittensor's Subnet 9 (SN9) demonstrated that a distributed network of incentivized, permissionless actors could each pretrain large language models (LLMs) ranging from 700 million to 14 billion parameters, while surpassing established baselines. While that work validated blockchain-based decentralized pretraining as viable, it contained core issues: (i) every miner had to fit an entire model locally, and (ii) "winner-takes-all" rewards encouraged model hoarding. Here we introduce IOTA (Incentivized Orchestrated Training Architecture), an architecture that addresses these limitations by transforming SN9's previously isolated competitors into a single cooperating unit that can scale arbitrarily while still rewarding each contributor fairly. Key preliminary results: (1) Data- and Pipeline-parallel SWARM architecture - An orchestrator distributes model layers across heterogeneous miners and streams activations between them, enabling model sizes to scale with the number of participants rather than being constrained by the VRAM of a single machine; (2) Granular, continuous incentives - Validators measure each miner's contribution and allocate token emissions proportionally; (3) Activation compression - We used model-bottlenecks to cut communication bandwidths of activations by up to 128x, vastly improving training speed; (4) Butterfly All-Reduce - Miners average disjoint parameter slices in O(1) bandwidth, offering linear scalability, redundancy and built-in collusion detection; (5) CLASP (Contribution Loss Assessment via Sampling of Pathways) - A fair attribution scheme assigns credit to miners proportional to their marginal utility and detects exploits, even when contributions are interdependent across the pipeline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。