arXiv:2602.01058cs.LGcs.AI2026-02被引 8

优化SFT时考虑后续强化学习,能显著提升模型最终表现。

Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning

  • 用重要性采样重加权SFT损失,缓解数据分布与强化学习不匹配问题。
  • 在AIME2025上使通过率提升最高达14.6%,验证了对后续RL的更好准备效果。
  • 方法轻量,可无缝集成到现有训练流程,适合长期迭代的模型开发团队。

推理型大模型的后训练通常包含离线监督微调(SFT)和在线强化学习(RL)两个阶段。然而,当前SFT常被孤立优化以最大化自身性能。我们发现,在相同的RL训练后,从强SFT检查点初始化的模型反而显著弱于从弱检查点初始化的模型。这归因于当前SFT-RL流程中的典型偏差:离线SFT数据的生成分布与在线RL中由自身回放策略优化的分布存在显著差异。为此,我们提出PEAR(基于策略评估的离线学习损失重加权算法),一种SFT阶段的方法,通过重要性采样对损失进行重加权,提供三种粒度(词元、块、序列)的变体。该方法可与标准SFT目标结合使用,仅需预先计算离线数据的概率,训练开销极低。我们在Qwen 2.5/3及DeepSeek-distilled模型上,针对可验证推理游戏和数学推理任务进行了控制实验。结果表明,PEAR在所有测试中均优于标准SFT,AIME2025上的通过率提升最高达14.6%。研究证明,将下游强化学习纳入考量来设计和评估SFT,是实现更整体化大模型后训练的关键一步。

原文摘要 · Abstract (English)

Post-training of reasoning LLMs is a holistic process that typically consists of an offline SFT stage followed by an online reinforcement learning (RL) stage. However, SFT is often optimized in isolation to maximize SFT performance alone. We show that, after identical RL training, models initialized from stronger SFT checkpoints can significantly underperform those initialized from weaker ones. We attribute this to a mismatch typical in current SFT-RL pipelines: the distribution that generates the offline SFT data can differ substantially from the policy optimized during online RL, which learns from its own rollouts. We propose PEAR (Policy Evaluation-inspired Algorithm for Offline Learning Loss Re-weighting), an SFT-stage method that corrects this mismatch and better prepares the model for RL. PEAR uses importance sampling to reweight the SFT loss, with three variants operating at the token, block, and sequence levels. It can be used to augment standard SFT objectives and incurs little additional training overhead once probabilities for the offline data are collected. We conduct controlled experiments on verifiable reasoning games and mathematical reasoning tasks on Qwen 2.5 and 3 and DeepSeek-distilled models. PEAR consistently improves post-RL performance over canonical SFT, with pass at 8 gains up to a 14.6 percent on AIME2025. Our results suggest that PEAR is an effective step toward more holistic LLM post-training by designing and evaluating SFT with downstream RL in mind rather than in isolation.

大模型训练监督微调强化学习SFT优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。