提出轻量级控制器GNMR，提升低精度语言模型训练稳定性。
GNMR: Runtime Stability Control for Low-Precision Large Language Model Training
- 通过梯度范数与历史均值比值动态监控运行时风险
- 在不改变数值格式下实现稀疏且预算可控的恢复操作
- 适用于多种量化训练场景，兼容不同后端
低精度语言模型训练中的稳定性是关键瓶颈：即使高效低成本路径也可能在少数算子上引发短暂数值风险。本文将此问题建模为运行时稳定性控制，提出轻量级控制器梯度范数-均值比（GNMR），通过比较可恢复单元当前梯度范数与其历史均值来判断风险。结合Δ-GNMR应对突发短时波动，GNMR在硬性maxO预算和短锁定期约束下，将局部风险信号映射为有界恢复动作，无需改变数值格式、核函数或后端方案。在激活量化压力测试、DeepSeek风格的配方级训练以及LLaMA-2 13B微调中，GNMR均保持高质量输出，且恢复操作稀疏且符合预算。结果表明，GNMR是一种后端无关的控制器，可在保障低成本执行的同时提升低精度训练稳定性。
原文摘要 · Abstract (English)
Training stability is a key bottleneck in low-precision language model training: efficient low-cost paths can still produce short-lived numerical risks at a small set of operators. We formulate this as runtime stability control and present Gradient Norm-to-Mean Ratio (GNMR), a lightweight controller that compares each recoverable unit's current gradient norm with its historical mean. Together with $Δ$-GNMR for abrupt short-window increases, GNMR maps local risk signals to bounded recovery actions under a hard $\mathrm{maxO}$ budget and a short lock interval, without changing the numerical format, kernel, or backend recipe. Across activation-quantization stress, DeepSeek-style recipe-level training, and LLaMA-2 13B fine-tuning, GNMR preserves high-fidelity quality with sparse, budgeted recovery. These results support GNMR as a backend-agnostic controller to improve low-precision training stability while preserving low-cost execution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。