实测大模型训练中硬件隐性错误的影响，发现其可导致模型收敛到不同结果。
Understanding Silent Data Corruption in LLM Training
- 通过对比健康与故障节点，用确定性执行隔离分析硬件隐性错误
- 部分节点的错误导致梯度扰动小但训练损失突增，模型收敛至不同最优解
- 为大规模模型训练中的可靠性问题提供关键实证依据，适合系统与模型研究者
随着大语言模型（LLM）训练规模扩大，硬件产生的隐性数据错误（SDC）日益突出，即计算出错但无明显失败信号。本文首次通过对比健康生产节点与存在SDC的故障节点，研究真实世界中SDC对LLM训练的影响。借助云平台获取被自动运维系统剔除的故障节点，利用XLA编译器的确定性执行及自研同步机制，从子模块计算、单个优化器步骤到整个训练周期三个层面分析误差影响。结果显示，不同故障节点的计算扰动差异显著；多数情况下子模块计算和梯度扰动较小，但部分节点仍会导致模型收敛至不同权重的最优解，甚至引发训练损失突增。该分析为理解并缓解SDC影响提供了重要洞见。
原文摘要 · Abstract (English)
As the scale of training large language models (LLMs) increases, one emergent failure is silent data corruption (SDC), where hardware produces incorrect computations without explicit failure signals. In this work, we are the first to investigate the impact of real-world SDCs on LLM training by comparing model training between healthy production nodes and unhealthy nodes exhibiting SDCs. With the help from a cloud computing platform, we access the unhealthy nodes that were swept out from production by automated fleet management. Using deterministic execution via XLA compiler and our proposed synchronization mechanisms, we isolate and analyze the impact of SDC errors on these nodes at three levels: at each submodule computation, at a single optimizer step, and at a training period. Our results reveal that the impact of SDCs on computation varies on different unhealthy nodes. Although in most cases the perturbations from SDCs on submodule computation and gradients are relatively small, SDCs can lead models to converge to different optima with different weights and even cause spikes in the training loss. Our analysis sheds light on further understanding and mitigating the impact of SDCs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。