TetraJet-v2实现4比特大模型训练，有效抑制权重震荡并控制异常值。
TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Control
- 采用双块无偏量化与NVFP4格式，提升4比特训练收敛性。
- 在370M参数模型上训练2120亿词，性能比基线提升51.3%。
- 适合追求高效低精度训练的算法工程师和部署研究者。
大语言模型训练成本高昂，推动了全量化训练（FQT）的发展。尽管新型4比特格式如NVFP4带来显著效率提升，但实现接近无损的低精度训练仍具挑战。本文提出TetraJet-v2，一种端到端的4比特全量化训练方法，适用于所有线性层中的激活、权重与梯度。我们识别出两个关键问题：权重震荡与异常值。为此，提出：1）无偏双块量化方法，使NVFP4线性层在大模型训练中实现近似最优收敛；2）OsciReset，首个有效抑制大模型权重震荡瓶颈的算法；3）OutControl，一种混合精度算法以保留异常值精度。TetraJet-v2在高达370M参数的模型上进行长达2120亿词的FP4预训练，平均将性能差距缩小至BF16的51.3%，同时相比FP8实现1.67倍的端到端加速。代码已开源：https://github.com/thu-ml/TetraJet-v2-NVFP4Training。
原文摘要 · Abstract (English)
Large Language Models (LLMs) training is prohibitively expensive, driving interest in low-precision fully-quantized training (FQT). While novel 4-bit formats like NVFP4 offer substantial efficiency gains, achieving near-lossless training at such low precision remains challenging. We introduce TetraJet-v2, an end-to-end 4-bit FQT method that leverages NVFP4 for activations, weights, and gradients in all linear layers. We identify two critical issues hindering low-precision LLM training: weight oscillation and outliers. To address these, we propose: 1) an unbiased double-block quantization method for NVFP4 linear layers with practically optimal convergence in LLM training, 2) OsciReset, the first effective algorithm to suppress LLMs' weight oscillation bottleneck, and 3) OutControl, a mix-precision algorithm to retain outlier accuracy. TetraJet-v2 outperforms prior methods on FP4 pre-training for LLMs across models up to 370M parameters trained up to 212B tokens, reducing the performance gap to BF16 by an average of 51.3% while enabling an 1.67x end-to-end speedup over FP8. The code is available at https://github.com/thu-ml/TetraJet-v2-NVFP4Training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。