提出统一4位训练新方案UFP4,解决现有方法的系统性精度损失问题。
Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe

- 发现非均匀格式E2M1存在几何偏置,导致误差逐层累积
- UFP4在多个模型上实现更低的精度损失,优于主流E2M1方案
- 适合追求高效低精度大模型训练的研究者与工程师
FP4训练可显著降低大模型预训练的内存与计算开销,但当前硬件方案(如NVIDIA Blackwell/Rubin级、AMD MI350系列)仍以E2M1数据格式为主。本研究揭示该选择的根本缺陷:E2M1等非均匀格式因表示区间几何不对称,存在固有的收缩偏置(Shrinkage Bias),即系统性负舍入误差。该偏差在多层传播中呈乘法累积,并被随机哈达玛变换(RHT)放大,统一解释了现有E2M1方案中的训练不稳定性。相比之下,均匀网格格式(如E1M2/INT4)避免了网格几何误差,能更有效将RHT带来的桶利用率提升转化为更高量化质量。基于此,我们提出UFP4——一种统一4位训练方案,对所有三个训练GEMM应用RHT,仅对dY启用随机舍入。在Dense 1.5B、MoE 7.9B和MoE 124B长周期预训练中,UFP4始终实现低于强基线的BF16相对损失,经缩放律分析与消融实验验证。结果表明,未来加速器应将E1M2/INT4类均匀4位网格作为首级训练原语支持。
原文摘要 · Abstract (English)
FP4 training promises substantial reductions in memory and computation cost for LLM pretraining, yet current FP4 hardware paths and recipes, including NVIDIA Blackwell/Rubin-class systems and AMD MI350-series GPUs, remain centered on E2M1 data elements. In this study, we identify a fundamental limitation of that choice: non-uniform formats such as E2M1 inherently suffer from Shrinkage Bias, a systematic negative rounding error caused by the geometric asymmetry of their representable bins. We show that this bias accumulates multiplicatively across layers and is amplified by the Random Hadamard Transform (RHT), providing a unified explanation for the training instability observed in existing E2M1-based FP4 recipes. In contrast, uniform grids (E1M2/INT4) bypass this grid-geometry error and better convert the improved bucket utilization from RHT into higher quantization quality. Based on this finding, we propose UFP4, a uniform 4-bit training recipe that applies RHT to all three training GEMMs while restricting stochastic rounding to dY alone. On Dense 1.5B, MoE 7.9B, and MoE 124B long-run pretraining, UFP4 consistently achieves lower BF16-relative loss degradation than strong E2M1-based baselines, supported by scaling-law analysis and ablation studies. Our results suggest that future accelerators should support E1M2/INT4-style uniform 4-bit grids as first-class training primitives alongside E2M1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。