全FP4训练可让大模型高效又准确，无需混合精度回退。
Quartet: Native FP4 Training Can Be Optimal for Large Language Models
- 全链路使用FP4低精度计算，无混合精度依赖。
- 在Llama模型上实现接近FP16的精度，显存占用减少50%。
- 专为NVIDIA Blackwell架构优化，适合追求极致效率的研究者。
直接以低精度训练大语言模型可提升吞吐量和能效，降低计算成本。NVIDIA最新Blackwell架构支持使用FP4变体进行极低精度运算。然而，现有FP4训练算法常导致显著精度下降，仍需依赖混合精度回退。本文研究硬件支持的FP4训练，提出一种端到端全FP4训练方法，所有主要计算（如线性层）均在低精度下完成。通过对Llama类模型的广泛评估，我们揭示了一条新的低精度缩放定律，量化了不同位宽与训练设置下的性能权衡。基于此发现，我们设计出在精度与计算间达到最优平衡的技术——Quartet。通过针对Blackwell架构优化的CUDA内核实现，验证了全FP4训练可媲美FP16半精度及FP8训练。代码已开源：https://github.com/IST-DASLab/Quartet。
原文摘要 · Abstract (English)
Training large language models (LLMs) models directly in low-precision offers a way to address computational costs by improving both throughput and energy efficiency. For those purposes, NVIDIA's recent Blackwell architecture facilitates very low-precision operations using FP4 variants. Yet, current algorithms for training LLMs in FP4 precision face significant accuracy degradation and often rely on mixed-precision fallbacks. In this paper, we investigate hardware-supported FP4 training and introduce a new approach for accurate, end-to-end FP4 training with all the major computations (i.e., linear layers) in low precision. Through extensive evaluations on Llama-type models, we reveal a new low-precision scaling law that quantifies performance trade-offs across bit-widths and training setups. Guided by this investigation, we design an "optimal" technique in terms of accuracy-vs-computation, called Quartet. We implement Quartet using optimized CUDA kernels tailored for Blackwell, demonstrating that fully FP4-based training is a competitive alternative to FP16 half-precision and to FP8 training. Our code is available at https://github.com/IST-DASLab/Quartet.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。