用MXFP8量化训练大模型，效率提升且不降精度。
Recipes for Pre-training LLMs with MXFP8
- 采用MXFP8-E4M3格式与特定转换算法实现高效训练。
- 80亿参数模型在15万亿词数据上训练，精度媲美BF16。
- 适合追求训练效率的模型开发者与硬件优化研究者。
在预训练中使用更少比特表示模型参数和相关张量已成为提升GPU效率而不损失精度的必要技术。NVIDIA Blackwell架构引入的微缩放（MX）格式在此领域取得重大进展，使窄浮点数据类型与每块更细粒度的缩放因子结合成为可能。这不仅实现了比以往方法更多的张量量化，还提升了这些张量操作的执行效率。有效利用MX格式需谨慎选择多种参数。本文回顾了这些选择，并展示MXFP8-E4M3数据类型与特定数值转换算法可使训练结果达到BF16水平。实验使用最大达80亿参数的模型，在最高15万亿词的高质量数据集上进行训练。
原文摘要 · Abstract (English)
Using fewer bits to represent model parameters and related tensors during pre-training has become a required technique for improving GPU efficiency without sacrificing accuracy. Microscaling (MX) formats introduced in NVIDIA Blackwell generation of GPUs represent a major advancement of this technique, making it practical to combine narrow floating-point data types with finer granularity per-block scaling factors. In turn, this enables both quantization of more tensors than previous approaches and more efficient execution of operations on those tensors. Effective use of MX-formats requires careful choices of various parameters. In this paper we review these choices and show how MXFP8-E4M3 datatype and a specific number conversion algorithm result in training sessions that match those carried out in BF16. We present results using models with up to 8B parameters, trained on high-quality datasets of up to 15T tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。