提出轻量级机制检测分布式训练中层间通信异常，保障大规模模型安全训练。
SENTINEL: Stagewise Integrity Verification for Pipeline Parallel Decentralized Training
- 用指数移动平均监控层间激活与梯度传输，无计算冗余。
- 在176个不可信节点上成功训练40亿参数大模型，保持收敛性。
- 首次为流水线并行提供理论保证，适合高风险分布式训练场景。
去中心化训练在跨不可信地理分布节点执行时面临严峻安全风险。现有拜占庭容错方法虽能应对数据并行（DP）训练，但流水线并行（PP）因模型层分置于不同工作节点，激活值与梯度在阶段间流动而非聚合，传统方法失效。我们提出SENTINEL，一种无需计算冗余的流水线并行完整性验证机制。该方法采用轻量级基于动量的监控，利用指数移动平均（EMAs）检测受损的阶段间通信。不同于传统针对副本梯度聚合的拜占庭容错方案，本方法验证层间连续激活/梯度传输的正确性。我们提供了该新设置下的理论收敛保证，在退化为标准训练时恢复经典收敛速率。实验表明，可在多达176个不可信节点的分布式环境中成功训练高达40亿参数的大规模语言模型（LLMs），同时保持模型收敛与性能。
原文摘要 · Abstract (English)
Decentralized training introduces critical security risks when executed across untrusted, geographically distributed nodes. While existing Byzantine-tolerant literature addresses data parallel (DP) training through robust aggregation methods, pipeline parallelism (PP) presents fundamentally distinct challenges. In PP, model layers are distributed across workers where the activations and their gradients flow between stages rather than being aggregated, making traditional DP approaches inapplicable. We propose SENTINEL, a verification mechanism for PP training without computation duplication. SENTINEL employs lightweight momentum-based monitoring using exponential moving averages (EMAs) to detect corrupted inter-stage communication. Unlike existing Byzantine-tolerant approaches for DP that aggregate parameter gradients across replicas, our approach verifies sequential activation/gradient transmission between layers. We provide theoretical convergence guarantees for this new setting that recovers classical convergence rates when relaxed to standard training. Experiments demonstrate successful training of up to 4B-parameter LLMs across untrusted distributed environments with up to 176 workers while maintaining model convergence and performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。