FP4训练大模型易发散，根源在权重梯度量化
Pretraining large language models with MXFP4 on Native FP4 Hardware
- 逐项开启FP4前向、激活梯度与权重梯度量化，控制变量分析
- 权重梯度量化导致收敛恶化，前向与激活梯度影响较小
- 确定性哈达玛旋转可稳定训练，随机方法无效
为何全流水线FP4训练大语言模型常出现发散，即使前向激活和激活梯度保持稳定？我们通过控制实验研究Transformer训练中的MXFP4量化，逐步启用前向传播（Fprop）、激活梯度（Dgrad）和权重梯度（Wgrad）的FP4，其余条件保持不变。在C4数据集上对Llama 3.1-8B进行完整预训练时发现，权重梯度量化是导致收敛退化的主因；仅前向和激活梯度使用FP4则仅增加少量额外训练令牌。通过结构化与随机干预实验，我们发现随机舍入和随机哈达玛旋转无法在权重梯度量化后稳定训练，而确定性哈达玛旋转能持续恢复优化稳定性。结果表明，训练不稳定性源于敏感梯度路径上的结构性微尺度误差，而非随机性不足。实验基于AMD Instinct MI355X GPU的原生MXFP4支持，避免软件模拟干扰。
原文摘要 · Abstract (English)
Why does full-pipeline FP4 training of large language models often diverge, even when forward activations and activation gradients remain stable? We address this question through a controlled study of MXFP4 quantization in transformer training, progressively enabling FP4 across forward propagation (Fprop), activation gradients (Dgrad), and weight gradients (Wgrad) while holding all other factors fixed. In full pretraining of Llama 3.1-8B on the C4 dataset, we observe that quantizing Wgrad is the primary driver of convergence degradation, whereas FP4 in Fprop and Dgrad alone introduces only modest additional token requirements. To interpret this behavior, we evaluate both structured and stochastic interventions under a controlled experimental setting. We find that stochastic rounding and randomized Hadamard rotations fail to stabilize training once Wgrad is quantized, whereas deterministic Hadamard rotations consistently restore stable optimization. These results suggest that FP4 training instability is driven by structured micro-scaling errors along sensitive gradient paths, rather than by insufficient stochasticity. We run experiments with native MXFP4 support on AMD Instinct MI355X GPUs, enabling controlled investigation of these effects without reliance on software emulation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。