RL微调能修复SFT导致的分布外遗忘问题
RL Fine-Tuning Heals OOD Forgetting in SFT
- 通过检查点分析发现SFT会早期遗忘OOD能力
- RL微调可恢复部分丢失的OOD性能，但仅限特定SFT阶段
- 关键在控制奇异向量旋转，适合研究模型鲁棒性者
监督微调（SFT）后接强化学习（RL）是提升大语言模型推理能力的标准后训练流程，但其原理尚不明确。我们通过逐检查点分析分布内（ID）与分布外（OOD）推理表现，发现OOD性能常在SFT早期达到峰值后下降，而ID性能持续提升。RL通常无法超越此早期峰值，而是恢复后期SFT中丢失的OOD能力，且仅对特定范围的SFT检查点有效。谱分析显示，这种遗忘与恢复模式与奇异向量旋转相关，而奇异值基本稳定。结果表明：SFT可能遗忘，RL可恢复，控制奇异向量旋转或能提升OOD鲁棒性。代码已公开于 https://github.com/jinhangzhan/RL_Heals_SFT.git。
原文摘要 · Abstract (English)
Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL) is a standard post-training recipe for improving Large Language Models (LLM) reasoning, but why it works remains unclear. We revisit the common claim that ``SFT memorizes, RL generalizes'' through checkpoint-wise analyses of in-distribution (ID) and out-of-distribution (OOD) reasoning. We find that OOD performance often peaks early during SFT and then declines despite continued improvement in ID reasoning. RL typically does not surpass this early SFT peak; rather, it restores OOD capability lost during later SFT, and only from a bounded range of SFT checkpoints. Further spectral analysis shows that this forgetting-and-recovery pattern correlates with rotations of singular vectors, while singular values remain largely stable. These findings suggest a more precise view of post-training dynamics: SFT can forget, RL can recover, and controlling singular-vector rotation may improve OOD robustness. Code is available at \href{https://github.com/jinhangzhan/RL\_Heals\_SFT.git}{https://github.com/jinhangzhan/RL\_Heals\_SFT}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。