对比强化学习与监督微调，发现前者提升旧能力，后者替换旧技能。
Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them
- 在相同模型上比较强化学习与监督微调的训练动态。
- 强化学习仅小幅提升数学能力，监督微调导致知识型任务性能下降更明显。
- 监督微调更改变中层MLP参数，可能引发跨领域性能退化，适合关注模型可塑性研究者。
通过在相同数学问题和模型上对比强化学习（RL）与监督微调（SFT）的训练动态，我们发现RL在数学任务上带来微小的域内提升，而在知识密集型基准如MMLU上略有退化；而这些趋势在SFT中更为显著。分析模型参数变化发现，两者均主要修改查询和键权重，但SFT更新幅度更大，并更影响中层MLP，我们据此推测这可能导致了跨域性能下降。为此我们尝试在训练中冻结部分模型结构以缓解性能退化，结果不明确：在GPQA:Diamond上有收益，但在其他基准上仍出现退化。综合来看，这些观察初步揭示了为何RL能放大现有能力，而SFT则会替换旧技能。
原文摘要 · Abstract (English)
Training large language models (LLMs) for reasoning via maths and code datasets has become a major new focus in LLM post-training. Two particularly popular approaches are reinforcement learning (RL) and supervised fine-tuning (SFT), but their training dynamics are poorly understood. We present a comparative analysis of RL and SFT on the same maths problems with the same model and similar hyperparameters. We find that RL yields minor in-domain gains on maths and slight degradation on knowledge-intensive benchmarks like MMLU, while both trends are more pronounced in SFT. We also analyse model parameters across checkpoints, observing that both algorithms modify query and key weights the most. Meanwhile, SFT exhibits greater updates and also affects mid-layer MLPs more, leading us to hypothesise that this may have caused the out-of-domain degradation. We therefore investigate whether freezing parts of the model during training can mitigate the reduced performance on knowledge-intensive benchmarks. However, our results are inconclusive, with benefits on GPQA:Diamond and degradation on other benchmarks. Taken together, our observations provide a preliminary indication for why RL amplifies existing capabilities, while SFT replaces old skills with new ones.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。