arXiv:2608.06946cs.DCcs.LG2026-08

提出无需令牌的流式学习协议,实现高效公平的分布式模型训练。

Stream Learning: Partition-Fair Gossip Learning Without Tokens

  • 以视频分块类比模型分区,按分区年龄选择传输内容。
  • 随机选邻居传最欠训分区,在无故障时性能媲美现有最优方法。
  • 在30%节点永久失效下表现更优,适合异构网络场景。

在去中心化协同模型训练中,节点通过反复交换本地模型部分实现共同优化。当前最优方案PTGL将权重矩阵分为S个固定分区,借助基于令牌的公平机制与邻居元数据交换传播。本文借鉴点对点直播思想,将模型分区视为视频分块,分区年龄对应分块稀缺性,提出两阶段选择策略的设计空间,并构建十种具体协议统称为流式学习(Stream Learning)。核心发现:最简协议Ri——向随机邻居发送本地最欠训分区——在无故障情况下性能等同于PTGL,且无需令牌计数和元数据交换。在30%最优节点永久崩溃的对抗性场景下,Ri在所有完整图配置中表现匹配或超越PTGL,异构度最高时(Dirichlet β=0.1),在HAR和MNIST上分别领先5.53%和5.41%。实验表明,仅依赖分区年龄的局部规则即可实现公平性,而基于令牌的速率控制与效用最大化反而在异构环境下低于该规则。

原文摘要 · Abstract (English)

In gossip learning, a network of nodes trains a shared model collaboratively, without a central coordinator, by repeatedly exchanging parts of their local models. The state-of-the-art protocol, Partitioned Token Gossip Learning (PTGL) of Heged{ü}s et al., splits the weight matrix into S fixed partitions and disseminates them using a token-based fairness mechanism coupled with per-neighbor metadata exchange. We revisit partition scheduling by analogy with peer-to-peer live streaming, where model partitions act as video chunks and partition age acts as chunk scarcity. The analogy yields a design space of two-stage selection strategies (partition first, or neighbor first), from which we instantiate ten concrete protocols collectively called Stream Learning. Our main finding is that the simplest of these protocols, which transmits the locally least-trained partition to a uniformly random neighbor (Ri), matches PTGL on fault-free workloads while requiring neither token counters nor metadata exchange. Under an adversarial 30% permanent crash of the best-performing nodes, Ri matches or outperforms PTGL across all complete-graph configurations tested, with the gap reaching 5.53% on HAR and 5.41% on MNIST in the most heterogeneous regime (Dirichlet $β$ = 0.1). In our experiments, partition fairness, captured by a single local rule on partition age, accounts for the gap; token-based rate control and utility maximization do not improve over this rule and, under heterogeneity, sit below it.

分布式学习去中心化公平性异构网络

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。