SFT与RL训练无法分离，交替进行会相互损害性能。
On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-training
- 证明先SFT后RL时，RL会使SFT损失上升
- 实验验证在Qwen3-0.6B上出现性能退化
- 适合关注大模型微调流程设计的研究者
大语言模型的后训练通常交替进行监督微调（SFT）和强化学习（RL）。二者目标不同：SFT最小化模型输出与专家回复之间的交叉熵损失，而RL则最大化来自人类偏好或规则验证器的奖励信号。现代推理模型普遍采用交替训练策略。然而，目前尚无理论解释二者是否可解耦。本文证明无论顺序如何，解耦均不可能：(1) 先SFT后RL时，在分布（基于KL）与景观（基于PL）分析下，RL均会增加SFT损失；(2) 先RL后SFT时，在类似条件下，SFT会降低RL所获得的奖励。在PL条件下，进一步推导出最优RL训练时长，识别决定RL能否提升SFT的非解耦阈值，并通过谱集中度界定梯度错位程度。Qwen3-0.6B上的实验确认了预测的性能下降，证实后训练中SFT与RL不可分离，否则将导致先前性能损失。
原文摘要 · Abstract (English)
Post-training of large language models routinely interleaves supervised fine-tuning (SFT) with reinforcement learning (RL). These two methods have different objectives: SFT minimizes the cross-entropy loss between model outputs and expert responses, while RL maximizes reward signals derived from human preferences or rule-based verifiers. Modern reasoning models have widely adopted the practice of alternating SFT and RL training. However, there is no theoretical account of whether they can be decoupled. We prove that decoupling is impossible in either order: (1) SFT-then-RL coupling: RL increases SFT loss under both distributional (KL-based) and landscape (PL-based) analyses; and (2) RL-then-SFT coupling: SFT lowers the reward achieved by RL under analogous conditions. Under the PL condition, we further derive the optimal RL duration that balances reward improvement against SFT degradation, identify the non-decoupling threshold governing when RL can improve SFT, and bound the gradient misalignment via spectral concentration. Experiments on Qwen3-0.6B confirm the predicted degradation, verifying that SFT and RL cannot be separated without loss of prior performance in the post-training pipeline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。