arXiv:2606.09932cs.LGcs.AI2026-06

解决大模型微调后强化学习失效问题,让模型重获可塑性。

When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoff

论文配图:When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoff
图 1 · 摘自论文原文
  • 通过融合基础模型与重置关键神经元,恢复过度微调后模型的可塑性
  • 在数学推理和智能体任务上,显著提升过微调模型的强化学习表现
  • 适合希望优化微调-强化学习流程的研究者和工程师

监督微调(SFT)后接强化学习(RL)已成为大语言模型后训练的标准流程。理想情况下,SFT应为RL提供有益的行为先验以进一步提升模型能力。然而,过度SFT的检查点在后续RL阶段常表现出改进有限。我们将其归因于模型可塑性的丧失:即初始策略难以被后续RL有效重塑。通过从参数变化、输出空间和强化学习优化动态等多角度分析发现,过度SFT模型倾向于产生过度自信的词元分布,并呈现尖锐的参数景观,导致RL优化困难。为此,我们提出 exttt{Rejuvenation},一种简单但有效的恢复方法,在保留有用SFT先验的同时恢复模型可塑性。该方法利用基于基础模型的融合减少过度微调带来的漂移,并通过针对性神经元重置缓解模型僵化。在数学推理和智能体任务上的实验表明,该方法能持续提升过训练SFT模型的强化学习性能,并增强对分布外任务的泛化能力。

原文摘要 · Abstract (English)

Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL) has become a standard pipeline for Large Language Model (LLM) post-training. SFT is expected to provide a useful behavioral prior for RL to further enhance model capabilities. However, checkpoints with excessive SFT often show limited improvement during RL. We attribute this failure to the loss of model plasticity: the reduced ability of an SFT-initialized policy to be effectively reshaped by subsequent RL. To better understand this phenomenon, we conduct detailed analysis from multiple perspectives, including parameter changes, output spaces, and RL optimization dynamics. Our results show that models from excessive SFT tend to produce over-confident token distributions and exhibit sharp parameter landscapes, which make them harder to optimize in the RL stage. To enable a more robust SFT-to-RL handoff, we propose \texttt{Rejuvenation}, a simple yet effective method that restores plasticity while preserving useful SFT-acquired priors. Rejuvenation leverages base-anchored model fusion to reduce excessive SFT-induced drift with targeted neuron reset to mitigate model rigidity. Experimental results on both math reasoning tasks and agentic tasks demonstrate that our approach consistently improves RL performance on over-trained SFT models, while also enhancing generalization to out-of-distribution tasks.

大模型训练强化学习模型可塑性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。