用FP4精度训练大模型，效率更高且不丢性能。
Towards Efficient Pre-training: Exploring FP4 Precision in Large Language Models
- 分模块分阶段用混合精度，精准控制不同层的量化精度。
- 在多个基准上达到BF16和FP8的准确率，计算开销更小。
- 适合追求极致效率的模型训练者，尤其硬件支持FP4时
大语言模型训练日益增长的算力需求催生了高效方法，如量化训练，通过低比特运算降低开销。尽管FP8已有潜力，但FP4因量化误差大、表示能力弱而难应用。基于Transformer架构，本文提出一种面向LLM的FP4训练方案，通过针对不同模块和训练阶段设计的混合精度策略克服上述挑战，使多头注意力和线性层得以合理处理。预训练方案通过细粒度量化方法与目标精度训练调度，保障反向传播稳定性。实验表明，该方案在准确性上可媲美BF16和FP8，理论计算成本更低。随着新一代支持FP4的硬件出现，本方法为超低精度训练奠定了基础。
原文摘要 · Abstract (English)
The burgeoning computational demands for training large language models (LLMs) necessitate efficient methods, including quantized training, which leverages low-bit arithmetic operations to reduce costs. While FP8 precision has shown potential, leveraging FP4 remains challenging due to inherent quantization errors and limited representation capability. Based on the Transformer architecture, we present an FP4 training scheme for LLMs, overcoming these obstacles through mixed-precision quantization strategies tailed for different modules and training stages. This allows us to apply the precision level suitable to distinct components within the model, ensuring that multi-head attention and linear layers are handled appropriately. Our pretraining recipe ensures stability in backpropagation by incorporating fine-grained quantization methods with a target precision training schedule. Experimental results demonstrate that our FP4 training scheme achieves accuracy comparable to BF16 and FP8, with smaller theoretical computational cost. With the advent of next-generation hardware supporting FP4, our method sets the foundation for efficient ultra-low precision training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。