通过模块机制提前预警大模型训练崩溃，比传统方法早数千步发现异常。
Mechanism-Driven Monitors for Preemptive Detection of LLM Training Instability
- 从关键模块功能出发设计内部监控信号
- 在损失发散前数千步即触发预警，低精度注意力效果显著
- 适合大规模训练稳定性保障，尤其对资源密集型场景
前沿大模型训练依赖海量加速器集群和长时间计算，一旦出现数值或超参数故障，训练可能持续数千步仍保持正常损失与梯度范数，导致严重代价。本文提出基于机制的训练不稳定性预检测方法，从各核心模块的功能角色及最早可能出现故障迹象的计算节点出发，推导内部监控指标。针对低精度Flash Attention，监控其QK双线性分解的谱熵，其一阶项在损失完全崩溃前已出现异常；对于MoE路由模块，根据其专家选择功能设计监测指标。在低精度注意力、大学习率及复合故障的故障注入实验中，这些信号对不同故障类型均呈现独特特征，在损失发散前数千步即可触发预警。
原文摘要 · Abstract (English)
Frontier large language model training consumes massive accelerator fleets and long wall-clock computation, making stability failures costly when they occur. After a numerical or a hyperparameter fault has already destabilized the training dynamics, it may continue for thousands of steps while loss and gradient norms still appear normal. We study mechanism-driven detection of training instability by deriving internal monitors from the functional role of each critical module and from the earliest computational sites where failures are expected to produce measurable signatures. For low-precision flash attention, we monitor the spectral entropy of a QK bilinear decomposition, whose first-order term becomes abnormal before the loss fully collapses. For MoE routers, we derive indicators from their role in expert selection. Our fault-injection experiments on low-precision attention, large learning-rate, and combined faults show that these signals provide distinct signatures for different failures, triggering thousands of steps before loss divergence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。