提出SeedFlood方法，实现大模型去中心化训练的高效通信。
SeedFlood: A Step Toward Scalable Decentralized Training of LLMs
- 利用零阶更新的可重构性，将消息压缩至近零大小。
- 通信开销几乎与模型规模无关，支持百亿参数模型跨百客户端训练。
- 在通信效率和泛化性能上优于传统方法，接近一阶方法表现。
本文提出一种新型去中心化训练方法SeedFlood，旨在实现大模型在复杂网络拓扑下的可扩展训练，并以极低通信开销达成全局共识。传统基于八卦（gossip）的方法存在消息通信成本随模型规模增长的问题，且信息在网络跳数中衰减导致全局共识效率低下。SeedFlood突破这一限制，利用零阶更新的可重构结构，使消息近乎为零大小，可在网络中快速洪泛至所有客户端。该机制使通信开销可忽略且与模型大小无关，彻底消除去中心化训练的主要瓶颈。实验表明，SeedFlood在去中心化LLM微调任务中持续优于基于八卦的基线方法，在泛化性能与通信效率上均占优，甚至在大规模场景下达到与一阶方法相当的结果。
原文摘要 · Abstract (English)
This work presents a new approach to decentralized training-SeedFlood-designed to scale for large models across complex network topologies and achieve global consensus with minimal communication overhead. Traditional gossip-based methods suffer from message communication costs that grow with model size, while information decay over network hops renders global consensus inefficient. SeedFlood departs from these practices by exploiting the seed-reconstructible structure of zeroth-order updates and effectively making the messages near-zero in size, allowing them to be flooded to every client in the network. This mechanism makes communication overhead negligible and independent of model size, removing the primary scalability bottleneck in decentralized training. Consequently, SeedFlood enables training in regimes previously considered impractical, such as billion-parameter models distributed across hundreds of clients. Our experiments on decentralized LLM fine-tuning demonstrate thatSeedFlood consistently outperforms gossip-based baselines in both generalization performance and communication efficiency, and even achieves results comparable to first-order methods in large scale settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。