纠正两个隐藏错误后，传统微调+强化学习方法超越了混合策略模型。
SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning
- 修复DeepSpeed和OpenRLHF中的梯度累积与损失加权缺陷
- 标准SFT-then-RL在数学基准上领先3.8至22.2个百分点
- 少量强化学习步骤即可超越复杂混合策略，更高效
近期混合策略优化方法通过交替或融合监督学习与强化学习信号，在大语言模型推理中报告了优于标准SFT-then-RL的性能。我们发现多篇已发表论文依赖存在缺陷的基线：DeepSpeed中因CPU卸载优化器导致梯度累积时无声丢弃中间micro-batch（影响TRL、OpenRLHF、Llama-Factory等框架），以及OpenRLHF中损失聚合方式错误地加权每个mini-batch损失。这两处错误显著抑制SFT表现，其中优化器问题贡献主要差距，损失聚合问题带来次要影响。修正后，标准SFT-then-RL在Qwen2.5-Math-7B上比所有评估的混合策略方法高3.8分，在Llama-3.1-8B上高22.2分。即使仅使用50步强化学习的简化版本，也优于混合策略方法，且计算量更低。
原文摘要 · Abstract (English)
Recent mixed-policy optimization methods for LLM reasoning that interleave or blend supervised and reinforcement learning signals report improvements over the standard SFT-then-RL pipeline. We show that numerous recently published research papers rely on a faulty baseline caused by two distinct bugs: a CPU-offloaded optimizer bug in DeepSpeed that silently drops intermediate micro-batches during gradient accumulation (affecting multiple downstream frameworks including TRL, OpenRLHF and Llama-Factory), and a loss aggregation bug in OpenRLHF that incorrectly weights per-mini-batch losses. Together they suppress SFT performance, with the optimizer bug accounting for most of the gap and the loss aggregation bug contributing a smaller additional effect. Once corrected, the standard SFT-then-RL pipeline surpasses every published mixed-policy method we evaluate by +3.8 points on math benchmarks with Qwen2.5-Math-7B and by +22.2 points with Llama-3.1-8B. Even a truncated variant with just 50 RL steps outperforms mixed-policy methods on math benchmarks while using fewer FLOPs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。