用谱对齐提前预警深度模型训练崩溃，比传统方法更早更准。
Spectral Alignment as Predictor of Loss Explosion in Neural Network Training
- 通过监测层输入与权重主奇异向量的分布对齐度来判断训练状态
- 谱对齐符号多样性下降可提前预测损失爆炸，比传统指标更灵敏
- 计算开销低，适合部署在实际训练中监控模型稳定性
深度神经网络训练中的损失爆炸可能导致数百万美元的训练投入化为乌有。传统的监控指标如权重和梯度范数往往滞后且不明确，其数值在不同模型甚至同一模型的不同层间差异巨大，难以建立统一的失败预警标准。本文提出一种新的、理论基础坚实的指标——谱对齐（Spectral Alignment, SA），用于监测层输入与权重矩阵主奇异向量之间的分布对齐情况。我们发现，该对齐的符号多样性坍塌是表征崩溃和训练发散的强大早期预警信号。在语言模型上的实验证明，监控SA分布能比传统标量指标更早、更清晰地发出损失爆炸的警告。由于其计算开销极低，SA是一种实用的模型训练保护工具。
原文摘要 · Abstract (English)
Loss explosions in training deep neural networks can nullify multi-million dollar training runs. Conventional monitoring metrics like weight and gradient norms are often lagging and ambiguous predictors, as their values vary dramatically across different models and even between layers of the same model, making it difficult to establish a unified standard for detecting impending failure. We introduce Spectral Alignment (SA), a novel, theoretically-grounded metric that monitors the distributional alignment between layer inputs and the principal singular vectors of weight matrices. We show that a collapse in the sign diversity of this alignment is a powerful early predictor of representational collapse and training divergence. Empirical results on language models demonstrate that monitoring the SA distribution provides a significantly earlier and clearer warning of loss explosions than traditional scalar metrics. SA's low computational overhead makes it a practical tool for safeguarding model training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。