用FP16替代BF16可解决大模型强化学习微调中的训练推理不一致问题
Defeating the Training-Inference Mismatch via FP16
- 将训练与推理的浮点精度统一为FP16，消除数值差异
- 在多个任务和框架下实现更稳定优化、更快收敛和更强性能
- 只需少量代码修改，无需改动模型结构或算法
大语言模型的强化学习微调常因训练与推理阶段的数值不一致导致不稳定。尽管已有工作通过算法修正或工程对齐尝试缓解此问题，我们发现其根源在于浮点精度本身。广泛使用的BF16虽动态范围大，但引入显著舍入误差，破坏了训练与推理的一致性。本文证明，仅将精度恢复为FP16即可有效消除该不匹配。该方法简单易行，现代框架原生支持，仅需少量代码修改，无需调整模型架构或学习算法。实验表明，统一使用FP16能带来更稳定的优化过程、更快的收敛速度，并在多种任务、算法和框架中实现更强性能。这些发现呼吁重新审视强化学习微调中的精度权衡。
原文摘要 · Abstract (English)
Reinforcement learning (RL) fine-tuning of large language models (LLMs) often suffers from instability due to the numerical mismatch between the training and inference policies. While prior work has attempted to mitigate this issue through algorithmic corrections or engineering alignments, we show that its root cause lies in the floating point precision itself. The widely adopted BF16, despite its large dynamic range, introduces large rounding errors that breaks the consistency between training and inference. In this work, we demonstrate that simply reverting to \textbf{FP16} effectively eliminates this mismatch. The change is simple, fully supported by modern frameworks with only a few lines of code change, and requires no modification to the model architecture or learning algorithm. Our results suggest that using FP16 uniformly yields more stable optimization, faster convergence, and stronger performance across diverse tasks, algorithms and frameworks. We hope these findings motivate a broader reconsideration of precision trade-offs in RL fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。