为大模型训练中的注意力机制设计了轻量级容错方案，有效检测并修复异常值。
ATTNChecker: Highly-Optimized Fault Tolerant Attention for Large Language Model Training
- 基于错误传播规律设计算法容错机制，专用于大模型注意力计算。
- 平均仅增加7%训练开销，可100%检测并修复极端错误值。
- 相比传统检查点恢复，故障恢复效率提升最高49倍，适合高可靠性训练场景。
大规模语言模型在自然语言处理任务中表现出色，但其训练过程计算密集且易受故障影响，尤其关注基于Transformer的模型中注意力机制的故障问题。本文通过系统性故障注入实验，研究了计算结果中INF、NaN及近INF值对训练的影响，发现这些错误会引发模型不可训练状态并中断训练流程，迫使从检查点重启。为此，提出首个针对大模型注意力机制的算法级容错（ABFT）方案ATTCChecker。该方案依据故障传播模式设计，兼顾系统可靠性与模型脆弱性，同时实现性能优化，提供轻量级保护以支持快速训练。在四个大模型上的评估显示，该方法平均仅引入7%训练开销，可完全检测并纠正所有极端错误；相比当前最优的检查点/恢复方法，恢复开销降低最多达49倍。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated remarkable performance in various natural language processing tasks. However, the training of these models is computationally intensive and susceptible to faults, particularly in the attention mechanism, which is a critical component of transformer-based LLMs. In this paper, we investigate the impact of faults on LLM training, focusing on INF, NaN, and near-INF values in the computation results with systematic fault injection experiments. We observe the propagation patterns of these errors, which can trigger non-trainable states in the model and disrupt training, forcing the procedure to load from checkpoints. To mitigate the impact of these faults, we propose ATTNChecker, the first Algorithm-Based Fault Tolerance (ABFT) technique tailored for the attention mechanism in LLMs. ATTNChecker is designed based on fault propagation patterns of LLM and incorporates performance optimization to adapt to both system reliability and model vulnerability while providing lightweight protection for fast LLM training. Evaluations on four LLMs show that ATTNChecker incurs on average 7% overhead on training while detecting and correcting all extreme errors. Compared with the state-of-the-art checkpoint/restore approach, ATTNChecker reduces recovery overhead by up to 49x.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。