无需调参或动态缩放，即可实现大模型高效FP8训练
$μ$nit Scaling: Simple and Scalable FP8 LLM Training
- 基于第一性原理分析，仅通过简单干预实现FP8训练
- 1B至13B参数模型全层FP8计算，性能媲美高精度基线
- 支持模型宽度间超参迁移，训练与推理数值一致
使用8位浮点数（FP8）格式进行大语言模型训练可显著提升效率，但数值精度降低使训练变得困难。当前方法需调整多种超参数、缩小模型规模或承受动态缩放因子计算开销。本文提出μnit Scaling（μS）方法，实现无需动态缩放因子或特殊超参数的简单且可扩展的FP8训练，适用于大规模模型。该方法基于对常见Transformer操作的第一性原理分析，仅需最少干预即可实现。我们在1B至13B参数模型上验证了该方法，所有隐藏层线性计算均在FP8下完成，达到与更高精度基线相当的模型质量，同时训练速度提升最多达33%。
原文摘要 · Abstract (English)
Large Language Model training with 8-bit floating point (FP8) formats promises significant efficiency improvements, but reduced numerical precision makes training challenging. It is currently possible to train in FP8 only if one is willing to tune various hyperparameters, reduce model scale, or accept the overhead of computing dynamic scale factors. We demonstrate simple, scalable FP8 training that requires no dynamic scaling factors or special hyperparameters, even at large model sizes. Our method, $μ$nit Scaling ($μ$S), also enables simple hyperparameter transfer across model widths, matched numerics across training and inference, and other desirable properties. $μ$nit Scaling is straightforward to implement, consisting of a set of minimal interventions based on a first-principles analysis of common transformer operations. We validate our method by training models from 1B to 13B parameters, performing all hidden linear layer computations in FP8. We achieve quality equal to higher precision baselines while also training up to 33% faster.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。