FP8-Flow-MoE让大模型训练无需频繁转换精度，提速21%还省内存。
FP8-Flow-MoE: A Casting-Free FP8 Recipe without Double Quantization Error
- 全链路使用FP8，仅保留2次显式类型转换
- 671B参数模型下吞吐提升21%，显存降低16.5GB
- 适合追求高效训练的大模型开发者
训练大规模混合专家（MoE）模型因计算与内存需求极高而成本巨大。尽管低精度训练可加速计算并减少内存占用，现有方法仍依赖以BF16为主的计算流程，频繁进行量化-反量化（Q/DQ）转换，削弱了FP8的理论效率。若直接移除这些转换，保持全流程在FP8中，则会引入双量化误差：沿不同维度量化的张量累积不一致的缩放因子，导致数值不稳定。本文提出FP8-Flow-MoE，一种量化一致的全FP8中心化数据流，包含缩放感知转置与融合式FP8算子，将显式类型转换从12次减少至2次。在671B参数的MoE模型上评估显示，相比BF16和朴素FP8基线，吞吐最高提升21%，单卡显存降低16.5GB，且收敛稳定。该方案已兼容TransformerEngine与Megatron-LM，即将开源。
原文摘要 · Abstract (English)
Training large Mixture-of-Experts (MoE) models remains computationally prohibitive due to their extreme compute and memory demands. Although low-precision training promises to accelerate computation and reduce memory footprint, existing implementations still rely on BF16-dominated dataflows with frequent quantize-dequantize (Q/DQ) conversions. These redundant casts erode much of FP8's theoretical efficiency. However, naively removing these casts by keeping dataflows entirely in FP8 introduces double quantization error: tensors quantized along different dimensions accumulate inconsistent scaling factors, degrading numerical stability. We propose FP8-Flow-MoE, an FP8 training recipe featuring a quantization-consistent FP8-centric dataflow with a scaling-aware transpose and fused FP8 operators that streamline computation and eliminate explicit cast operations from 12 to 2. Evaluations on a 671B-parameter MoE model demonstrate up to 21\% higher throughput and 16.5 GB lower memory usage per GPU compared to BF16 and naïve FP8 baselines, while maintaining stable convergence. We provide a plug-and-play FP8 recipe compatible with TransformerEngine and Megatron-LM, which will be open-sourced soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。