研究大模型训练中隐蔽数据错误的破坏力及应对方法。
Exploring Silent Data Corruption as a Reliability Challenge in LLM Training
- 通过注入GPU计算故障，分析不同阶段和位数的敏感性。
- 发现局部故障可引发损失突增、梯度异常和参数漂移。
- 提出轻量检测法，复算出错步骤可有效缓解影响。
随着大语言模型规模扩大，训练过程中的故障后果日益严重。本文聚焦于隐蔽数据错误（SDC）：一种绕过系统检测的硬件故障，可能表现为无害数值噪声，也可能导致有害梯度污染，引发损失突增、发散或训练停滞。本研究在受控环境下，针对大模型预训练过程中的间歇性SDC进行分析，通过在GPU矩阵乘指令层面注入故障，评估不同比特位、核函数和执行阶段的敏感性。结果表明，局部故障可引发显著污染，包括NaN传播、损失与梯度范数短时波动、注意力分数异常，以及持续的参数漂移。基于这些污染特征，我们提出一种轻量级检测方法，能识别潜在有害的参数更新。在包含60M、350M和1.3B参数的LLaMA模型上实验表明，检测后重算最近训练步可有效缓解此类事件的影响。
原文摘要 · Abstract (English)
As Large Language Models (LLMs) scale in size and complexity, the consequences of failures during training become increasingly severe. A major challenge arises from Silent Data Corruption (SDC): hardware-induced faults that bypass system-level detection mechanisms. SDC may behave like benign numerical noise, but can also cause harmful gradient corruption that leads to loss spikes, divergence, or stalled progress. This work provides a controlled study of how intermittent SDC affects LLM pretraining. Using targeted fault injection at the level of GPU matrix-multiply instructions, we characterize the sensitivity of different bit positions, kernel functions, and execution stages. Our analysis shows that locally originating faults can produce impactful corruption, including NaN propagation, short-lived spikes in loss, gradient norm, and attention logits, as well as persistent parameter divergence. Building on the observed corruption signatures, we propose a lightweight detection method that identifies potentially harmful parameter updates. Experiments on LLaMA models with 60M, 350M, and 1.3B parameters demonstrate that recomputing the most recent training step upon detection can effectively mitigate the impact of these events.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。