强化微调比监督微调更抗遗忘,适合持续学习场景。
Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Training
- 用强化微调替代监督微调,避免知识覆盖。
- 连续训练下,强化微调保持旧任务性能,接近多任务训练效果。
- 无需额外正则,机制天然防干扰,适合长期模型更新。
持续后训练(CPT)是适应基础模型(如多模态大语言模型)应对不断演进下游任务的流行有效方法。现有研究主要关注数据回放、模型扩展或参数正则化等策略,但对学习范式本身的影响尚未深入。本文对比了两种核心后训练范式:监督微调(SFT)与强化微调(RFT),探究其在CPT中对知识保留的影响。实验基于Qwen2.5-VL-7B-Instruct模型,在多个多模态任务上进行。结果表明:(1)连续学习时,SFT导致先前任务出现灾难性遗忘;而RFT能自然保留先验知识,性能接近多任务训练。(2)RFT不仅保护且提升模型在标准基准上的通用能力,而SFT严重削弱通用性。进一步分析发现,该稳定性并非源于KL惩罚或思维链等显式机制。通过对RFT学习动态的研究,发现其选择性更新机制可自然防止对已有知识的干扰。据此提出一种基于滚动评估的实例过滤算法(RIF-RFT),聚焦可学习样本以提升训练效率。研究表明,RFT是持续后训练的稳健范式。
原文摘要 · Abstract (English)
Continual post-training (CPT) is a popular and effective technique for adapting foundation models like multimodal large language models to ever-evolving downstream tasks. While existing research primarily focuses on methods like data replay, model expansion, or parameter regularization, the fundamental role of the learning paradigm remains largely unexplored. This paper presents a comparative analysis of two core post-training paradigms: supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT), investigating their respective impacts on knowledge retention during CPT. Our experiments are conducted across multiple multimodal tasks, utilizing Qwen2.5-VL-7B-Instruct as the base model. The investigation yields two significant findings: (1) When continuously learning on downstream tasks, SFT leads to catastrophic forgetting of previously learned tasks. In contrast, RFT inherently preserves prior knowledge and achieves performance comparable to multi-task training. (2) RFT successfully protects and even enhances the model's general knowledge on standard benchmarks, while SFT degrades general model capabilities severely. Further analysis reveals that this stability is not primarily due to explicit mechanisms like KL penalty or chain-of-thought reasoning. We investigate RFT's learning dynamics and find that its selective update mechanism inherently prevents interference with established knowledge. Based on this insight, we propose a rollout-based instance filtering algorithm (RIF-RFT) that enhances the training efficiency of RFT by focusing on learnable samples. Our comprehensive study demonstrates the superiority of RFT as a robust paradigm for continual post-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。