用4比特浮点数全程训练大模型，性能接近传统方法。
FP4 All the Way: Fully Quantized Training of LLMs
- 全链路使用FP4精度，权重、激活和梯度均量化。
- 在2000亿词数据上训练70亿参数模型，性能媲美半精度基准。
- 提出NVFP4格式与动态阈值机制，提升量化训练稳定性。
我们首次实现了大规模语言模型（LLM）的全量化训练（FQT），在最大达2000亿词的数据集上，主要采用4比特浮点（FP4）精度对权重、激活和梯度进行量化。系统研究了块大小、缩放格式和舍入方式等关键设计选择。分析表明，NVFP4格式（每16个FP4值共享一个E4M3表示的缩放因子）表现最优。反向传播和参数更新采用随机舍入，前向传播使用四舍五入以增强稳定性。我们还发现有效量化训练的理论与实证阈值：当梯度范数低于约√3倍量化噪声时，量化训练效果下降。基于此，我们在256个Intel Gaudi2加速器上成功训练了一个70亿参数模型。最终模型在下游任务上的表现与标准BF16基线相当，证明了FP4训练在大规模语言模型中的可行性与高效率。参考实现已公开于https://github.com/Anonymous1252022/fp4-all-the-way。
原文摘要 · Abstract (English)
We demonstrate, for the first time, fully quantized training (FQT) of large language models (LLMs) using predominantly 4-bit floating-point (FP4) precision for weights, activations, and gradients on datasets up to 200 billion tokens. We extensively investigate key design choices for FP4, including block sizes, scaling formats, and rounding methods. Our analysis shows that the NVFP4 format, where each block of 16 FP4 values (E2M1) shares a scale represented in E4M3, provides optimal results. We use stochastic rounding for backward and update passes and round-to-nearest for the forward pass to enhance stability. Additionally, we identify a theoretical and empirical threshold for effective quantized training: when the gradient norm falls below approximately $\sqrt{3}$ times the quantization noise, quantized training becomes less effective. Leveraging these insights, we successfully train a 7-billion-parameter model on 256 Intel Gaudi2 accelerators. The resulting FP4-trained model achieves downstream task performance comparable to a standard BF16 baseline, confirming that FP4 training is a practical and highly efficient approach for large-scale LLM training. A reference implementation is supplied in https://github.com/Anonymous1252022/fp4-all-the-way .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。