通过一致性投票定位大模型预训练中的故障源头，不依赖运行时监控。
SCOUT: Symmetric Consensus Outlier Detection for Failure Localization in LLM Pre-Training
- 用等价副本间严格多数共识识别异常节点
- 在训练挂起时仍能通过外置观察器响应，定位延迟与数据错误
- 无需修改代码即可集成主流训练框架，适合大规模训练故障排查
大模型预训练中，同步机制会将局部停滞、延迟和数值错误传播为全局症状，掩盖其真实来源。现有诊断方法依赖运行时监控，无法在训练阻塞或终止后报告；事后日志仅保存同步后的症状，丢失引发故障的工作负载与运行条件。我们提出 SCOUT，一个基于统一设计原则的实时故障定位框架：通过等价副本间的严格多数共识识别异常。SCOUT对齐副本进度、时间与数值证据，利用共识集体通信（C3）抽象检测与其他同伴签名不一致的进程。外置 CPU 观察器在训练挂起时仍可响应，而内嵌重放机制可在不影响主任务的情况下，重现导致慢节点和隐性数据损坏（SDC）的问题，保留模型状态、内核、内存分配、通信路径及热/内存压力。集体指纹暴露局部协议分歧，干净重放覆盖验证检查点数值完整性，防止从被 SDC 损坏的状态恢复。SCOUT 可无缝集成至 PyTorch、TorchTitan、Megatron-Core 与 DeepSpeed，无需修改训练循环或框架源码。开源地址：https://github.com/LMResiliency/lm-resiliency。
原文摘要 · Abstract (English)
In LLM pre-training, synchronization propagates rank-local stalls, slowdowns, and numerical errors into job-wide symptoms, obscuring their origin. Existing diagnosis often relies on in-process monitors that cannot report after the trainer blocks or terminates, or on post-mortem logs that preserve only synchronized symptoms; offline health tests lose the workload and operating conditions that triggered the failure. We present SCOUT, a unified runtime failure-localization framework built on one design principle: identify outliers through strict-majority consensus among equivalent replicas. SCOUT aligns replica progress, timing, and numerical evidence, then uses its Consensus Collective Communication (C3) abstraction to identify ranks whose compact signatures disagree with their peers. An out-of-band CPU observer remains responsive when training hangs, whereas in-situ replay exercises recurring stragglers and silent data corruption (SDC) beside the live job with its model state, kernels, allocations, communication path, and thermal and memory pressure present. Collective fingerprints expose rank-local protocol divergence. Clean replay coverage certifies checkpoint numerical integrity, preventing recovery from selecting state corrupted by SDC. SCOUT integrates with PyTorch, TorchTitan, Megatron-Core, and DeepSpeed without training-loop or framework-source modifications. SCOUT is open source at https://github.com/LMResiliency/lm-resiliency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。