提出首个系统性分析LLM训练中隐性数据错误的框架，有效提升训练鲁棒性。
TrainSDC: Characterizing and Mitigating Silent Data Corruption in Large Language Model Training
- 基于对前向与反向传播路径的误差传播机制分析，针对性设计保护策略。
- 在稀疏与密集故障注入下，保持近似无故障训练效果，运行开销仅1.65%-6.76%。
- 适用于大模型训练场景，尤其适合对可靠性要求高的生成式AI系统。
大型语言模型训练正面临日益严重的隐性数据错误（SDC）威胁，而现有防护方法多将Transformer计算视为整体，因对其脆弱性理解不足。本文首次系统性刻画了Transformer训练中前向与反向传播各主要计算接口的SDC脆弱性。分析发现：前向传播的脆弱性高度依赖位置，特别是Q/K路径上的故障会引发持续训练偏差；反向传播则主要受梯度指数分布影响，而非具体计算位置。基于此，我们提出TrainSDC——一种基于特征引导的防护框架，包含Q/K路径重计算、残差增益监控与指数感知梯度缩放。在Llama 3.2-1B和Qwen3-0.6B上的实验表明，该框架在稀疏与密集故障注入下仍能维持接近无故障执行的训练行为，运行开销仅为1.65%-6.76%。
原文摘要 · Abstract (English)
LLM training is increasingly vulnerable to silent data corruption (SDC), yet existing protection methods largely treat Transformer computations uniformly because their vulnerability remains poorly understood. We present the first systematic characterization of SDC vulnerability across major computation interfaces in both the forward and backward passes of Transformer training. Our analysis reveals two distinct error propagation mechanisms: forward-pass vulnerability is highly location dependent, with faults on the Q/K path producing persistent training deviations, whereas backward-pass vulnerability is largely governed by gradient exponent distributions rather than computation locations. Motivated by these observations, we propose TrainSDC, a characterization-guided protection framework consisting of Q/K-path recomputation, residual-gain monitoring, and exponent-aware gradient scaling. Experiments on Llama 3.2-1B and Qwen3-0.6B show that TrainSDC maintains training behavior close to fault-free execution under both sparse and dense fault injection while introducing only 1.65%-6.76% runtime overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。