ZClip动态调整梯度裁剪阈值，自动抑制大模型训练中的损失突增。
ZClip: Adaptive Spike Mitigation for LLM Pre-Training
- 基于z-score检测梯度异常，自适应调整裁剪阈值
- 有效防止损失尖峰，减少训练中断与人工干预
- 适合大规模语言模型训练，提升稳定性与效率
大语言模型（LLMs）训练面临梯度不稳定和损失突增等挑战，易导致灾难性发散，需频繁恢复检查点或跳过数据批次。传统梯度裁剪方法如固定阈值或范数裁剪依赖静态设定或启发式规则，无法有效应对动态变化的梯度规模，造成学习效率低下且需频繁人工干预。本文提出ZClip，一种基于梯度范数统计特性自适应调整裁剪阈值的算法。它通过z-score异常检测机制，主动识别并抑制大梯度突增，在不干扰正常收敛的前提下预防恶性损失峰值。该方法无需对梯度规模或时间演化做先验假设，可显著提升训练稳定性。代码已开源：https://github.com/bluorion-com/ZClip。
原文摘要 · Abstract (English)
Training large language models (LLMs) presents numerous challenges, including gradient instability and loss spikes. These phenomena can lead to catastrophic divergence, requiring costly checkpoint restoration and data batch skipping. Traditional gradient clipping techniques, such as constant or norm-based methods, fail to address these issues effectively due to their reliance on fixed thresholds or heuristics, leading to inefficient learning and requiring frequent manual intervention. In this work, we propose ZClip, an adaptive gradient clipping algorithm that dynamically adjusts the clipping threshold based on statistical properties of gradient norms over time. Unlike prior reactive strategies, ZClip proactively adapts to training dynamics without making any prior assumptions on the scale and the temporal evolution of gradient norms. At its core, it leverages z-score-based anomaly detection to identify and mitigate large gradient spikes, preventing malignant loss spikes while not interfering with convergence otherwise. Our code is available at: https://github.com/bluorion-com/ZClip.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。