全FP8计算实现大规模LLM训练，性能稳定且速度大幅提升
Towards Fully FP8 GEMM LLM Training at Scale
- 设计新架构，全程使用FP8进行Transformer块的矩阵运算
- 在大规模训练中实现更高吞吐，性能媲美传统BF16训练
- 监控关键指标可提前发现低精度训练中的发散风险
尽管FP8数据格式在大语言模型（LLM）预训练中具有巨大潜力，但其在大规模训练中的应用受限于稳定性问题。现有方法常依赖次优的细粒度FP8内核，或在注意力投影等敏感模块回退到高精度矩阵乘法（GEMM），牺牲了潜在的吞吐提升。本文首次提出一类新架构，可在前向与反向传播中对所有Transformer块内的GEMM均支持FP8计算。该设计显著提升大规模训练吞吐，同时保持与标准BF16训练相当的下游性能。通过减少大异常激活值，该架构促进长期稳定的FP8训练，并识别出用于监测低精度训练的关键指标，可预测未来可能的数值发散。
原文摘要 · Abstract (English)
Despite the significant potential of FP8 data formats for large language model (LLM) pre-training, their adoption has been limited due to challenges in maintaining stability at scale. Existing approaches often rely on suboptimal fine-grained FP8 kernels or fall back to higher-precision matrix multiplications (GEMMs) in sensitive components, such as attention projections, compromising potential throughput gains. We introduce a new class of LLM architectures that, for the first time, support FP8 computation for all GEMMs within transformer blocks during both forward and backward passes. This enables unprecedented throughput gains, particularly at scale, while matching the downstream performance of standard BF16 training. Our architecture design reduces large outlier activations, promoting stable long-term FP8 training. In addition, we identify key metrics to monitor low-precision training and predict potential future divergences.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。