首次实现大模型强化学习全链路4比特推理训练，显著提升精度。
HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models
- 采用分层缩放的HiFloat4格式与滚动残差量化修复激活值溢出问题。
- 在Qwen2.5-3B上将4比特精度损失从4.9%降至1.1%，接近全精度表现。
- 适用于开源MXFP4格式，适合追求极致压缩的部署场景。
本文首次实现端到端4比特（FP4）强化学习后训练，包括推理和训练策略的前向与反向传播均在4比特下运行。系统性研究表明，4比特强化学习性能下降主因并非训练侧量化误差，而是推理阶段激活值量化：异常值拉伸动态范围，导致大量激活值在4比特下下溢为零。反直觉的是，仅将训练策略恢复至高精度而保持推理为FP4时，准确率反而低于全FP4基线，暴露了推理与训练不匹配的核心问题，排除了传统预训练式修复方案。为此提出滚动残差量化（Rollout-ResQ）：仅在FP4推理矩阵乘法中添加一个硬件友好稀疏模式的残差修正项，轻量级修复因异常值导致的精度损失，无需增加计算开销。结合HiFloat4（HiF4）格式——其三级分层缩放在紧凑4比特预算下仍保持分辨率——在Qwen2.5-3B和Qwen2.5-Math-7B上，将与BF16的精度差距从4.9%缩小至1.1%，使完全量化4比特强化学习逼近全精度。应用于开源标准MXFP4时，差距也从13.6%降至5.3%，表明4比特格式选择是可恢复精度的决定性因素。结果确立了HiF4作为端到端4比特强化学习的关键格式，而Rollout-ResQ则是修复激活精度损失的有效机制。
原文摘要 · Abstract (English)
We present, to our knowledge, the first end-to-end FP4 RL post-training, in which both the rollout and training policies, including their forward and backward passes, operate at 4-bit precision. A systematic study reveals that the dominant source of degradation in FP4 RL is not training-side quantization error but rollout activation quantization: outliers stretch the dynamic range so far that a large number of activation values underflow to zero under FP4. Counterintuitively, restoring the training policy to higher precision while keeping the rollout in FP4 makes accuracy worse than full FP4 baseline, exposing rollout-training mismatch as the principal failure mode and ruling out standard pretraining-style fixes. We address this with Rollout Residual Quantization (Rollout-ResQ): a single residual correction term constrained to a hardware-friendly sparsity pattern, added only to the FP4 rollout matmul -- a lightweight correction that recovers most of the precision lost to outlier-driven underflow without inflating the rollout's compute footprint. On Qwen2.5-3B and Qwen2.5-Math-7B, Rollout-ResQ paired with the HiFloat4 (HiF4) format -- whose three-level hierarchical scaling preserves resolution under FP4's tight 4-bit budget -- closes the accuracy gap to BF16 from 4.9% to 1.1%, bringing fully quantized FP4 RL within striking distance of full precision. Applied to the open-standard MXFP4, the same recipe narrows the gap from 13.6% to 5.3%, revealing that FP4 format choice is a key factor that determines the ceiling on recoverable accuracy. Together, these results establish HiF4 as the enabling format for end-to-end FP4 RL post-training, and Rollout-ResQ as the activation-side mechanism that makes the gap to BF16 closable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。