强化学习比监督微调更少遗忘旧知识，因它偏好与原模型相似的解。
RL's Razor: Why Online Reinforcement Learning Forgets Less
- 用KL散度衡量分布偏移，发现RL更新更贴近原始模型
- 在语言和机器人模型上验证，RL遗忘程度显著低于SFT
- 适合关注模型持续学习与知识保留的研究者
对比基于强化学习（RL）与监督微调（SFT）的模型微调，尽管在新任务上表现相近，但RL显著更好保留了原有知识与能力。我们发现遗忘程度由分布偏移决定，以新任务上微调后策略与基础策略之间的KL散度衡量。分析表明，在策略上的RL隐式倾向于在解决新任务的所有可能解中，选择与原模型KL最小的解；而SFT可能收敛到与原模型任意远离的分布。我们在大语言模型和机器人基础模型上验证了这一发现，并进一步从理论上证明：基于策略的RL更新会导致更小的KL变化。我们提出这一原则为“RL的剃刀”：在所有解决新任务的方法中，RL偏好与原模型最接近的解。
原文摘要 · Abstract (English)
Comparison of fine-tuning models with reinforcement learning (RL) and supervised fine-tuning (SFT) reveals that, despite similar performance at a new task, RL preserves prior knowledge and capabilities significantly better. We find that the degree of forgetting is determined by the distributional shift, measured as the KL-divergence between the fine-tuned and base policy evaluated on the new task. Our analysis reveals that on-policy RL is implicitly biased towards KL-minimal solutions among the many that solve the new task, whereas SFT can converge to distributions arbitrarily far from the base model. We validate these findings through experiments with large language models and robotic foundation models and further provide theoretical justification for why on-policy RL updates lead to a smaller KL change. We term this principle $\textit{RL's Razor}$: among all ways to solve a new task, RL prefers those closest in KL to the original model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。