提出Quartet II，实现NVFP4下高效低误差大模型训练
Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimation
- 设计新型无偏量化方法MS-EDEN，误差低于随机舍入两倍
- 在380亿词上训练19亿参数模型，性能接近FP16
- 专为NVIDIA Blackwell GPU优化，速度最高提升4.2倍
NVIDIA Blackwell GPU支持的NVFP4低精度格式首次使大规模语言模型端到端全量化预训练成为可能。然而现有量化训练方法为获得更准确的无偏梯度估计，采用随机舍入（SR），牺牲了该格式的部分表征能力，导致相比标准FP16和FP8训练明显失准。本文提出一种针对微尺度格式的新型无偏量化方案MS-EDEN，其量化误差比SR降低超过2倍。将其集成至线性层的全新全NVFP4量化方案Quartet II中。理论分析表明，Quartet II在前向与反向传播的所有主要矩阵乘法中均实现更优的梯度估计。该方案还与近期专为NVFP4设计的训练优化技术良好协同。我们在380亿词数据上对最大19亿参数的LLM进行端到端训练验证。提供可在NVIDIA Blackwell GPU上运行的内核，相较BF16最快提速4.2倍。代码已开源。
原文摘要 · Abstract (English)
The NVFP4 lower-precision format, supported in hardware by NVIDIA Blackwell GPUs, promises to allow, for the first time, end-to-end fully-quantized pre-training of massive models such as LLMs. Yet, existing quantized training methods still sacrifice some of the representation capacity of this format in favor of more accurate unbiased quantized gradient estimation by stochastic rounding (SR), losing noticeable accuracy relative to standard FP16 and FP8 training. In this paper, improve the state of the art for quantized training in NVFP4 via a novel unbiased quantization routine for micro-scaled formats, called MS-EDEN, that has more than 2x lower quantization error than SR. We integrate it into a novel fully-NVFP4 quantization scheme for linear layers, called Quartet II. We show analytically that Quartet II achieves consistently better gradient estimation across all major matrix multiplications, both on the forward and on the backward passes. In addition, our proposal synergizes well with recent training improvements aimed specifically at NVFP4. We further validate Quartet II on end-to-end LLM training with up to 1.9B parameters on 38B tokens. We provide kernels for execution on NVIDIA Blackwell GPUs with up to 4.2x speedup over BF16. Our code is available at https://github.com/IST-DASLab/Quartet-II .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。