首次实现大模型FP4训练，显著降低计算成本。
Optimizing Large Language Model Training Using FP4 Quantization
- 设计可微量化估计器，精准更新权重
- 激活值异常抑制与补偿，防止崩溃
- 支持130亿参数模型，适配未来硬件
大语言模型训练的算力需求持续增长，量化训练通过低比特运算有望降低成本。尽管FP8已验证可行性，但FP4因量化误差大、表示能力弱仍难应用。本文提出首个面向LLM的FP4训练框架，创新性地引入可微量化估计器以实现精确权重更新，并采用异常值钳制与补偿策略防止激活值崩溃。为保障稳定性，框架融合混合精度训练和逐向量量化机制。实验表明，该框架在130亿参数模型上训练至1000亿词时，性能接近BF16与FP8，仅轻微下降，具备良好扩展性。随着新一代支持FP4的硬件出现，本工作为超低精度高效训练奠定基础。
原文摘要 · Abstract (English)
The growing computational demands of training large language models (LLMs) necessitate more efficient methods. Quantized training presents a promising solution by enabling low-bit arithmetic operations to reduce these costs. While FP8 precision has demonstrated feasibility, leveraging FP4 remains a challenge due to significant quantization errors and limited representational capacity. This work introduces the first FP4 training framework for LLMs, addressing these challenges with two key innovations: a differentiable quantization estimator for precise weight updates and an outlier clamping and compensation strategy to prevent activation collapse. To ensure stability, the framework integrates a mixed-precision training scheme and vector-wise quantization. Experimental results demonstrate that our FP4 framework achieves accuracy comparable to BF16 and FP8, with minimal degradation, scaling effectively to 13B-parameter LLMs trained on up to 100B tokens. With the emergence of next-generation hardware supporting FP4, our framework sets a foundation for efficient ultra-low precision training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。