arXiv:2506.23508cs.CLcs.AI2025-06中稿 · ICLR被引 14

对比微调方法发现,强化学习微调更不易遗忘旧知识。

Why Reinforcement Fine-Tuning Enables MLLMs Preserve Prior Knowledge Better: A Data Perspective

  • 用拼图任务测试模型对先验知识的保留能力
  • 强化微调比监督微调遗忘少,但学得慢
  • 数据分布方向与模型原概率格局一致时更稳定

后训练算法如监督微调(SFT)和强化微调(RFT)广泛用于适应多模态大模型至下游任务。尽管在任务适配上有效,其对先验知识保留的影响尚不明确。本文引入拼图这一预训练语料库中不存在的新任务,系统研究了SFT与RFT在开源Qwen2.5-VL系列上的表现。实验显示显著权衡:SFT快速掌握新任务但导致灾难性遗忘,而RFT学习较慢却更好保留先验知识。通过分析学习动态,发现RFT主要强化与基模型概率分布自然对齐的正确样本,减少对先验知识的干扰。此外,在模拟的RFT回溯轨迹上训练,因影响幅度更小且方向更一致,使SFT也能在快速学习的同时更好保留知识。该框架在数学与科学问答任务中验证一致,表明后训练数据分布而非算法差异是遗忘的关键,凸显RFT在持续后训练中的潜力。

原文摘要 · Abstract (English)

Post-training algorithms such as Supervised Fine-Tuning (SFT) and Reinforcement Fine-Tuning (RFT) are widely used to adapt (multimodal) large language models to downstream tasks. While effective at task adaptation, their impact on retaining prior knowledge remains unclear. In this paper, we introduce jigsaw puzzles as a novel task absent from existing pretraining corpora and systematically study the behavior of SFT and RFT on the open-source Qwen2.5-VL series. Our experiments reveal a sharp trade-off: SFT enables rapid task acquisition but leads to catastrophic forgetting, whereas RFT learns more slowly but better maintains prior knowledge. We study this phenomenon through learning dynamics by examining both the magnitude and direction of how training data influence prior knowledge. Our analysis shows that RFT mainly reinforces correct samples naturally aligned with the base model's probability landscape, leading to weaker interference with prior knowledge. Moreover, training on RFT-simulated rollouts, which exert a smaller magnitude of influence and are better aligned in direction to prior knowledge, allows SFT to preserve prior knowledge better while rapidly learning new tasks. We further validate our framework on Qwen2.5 post-training in math and scientific QA, observing consistent forgetting and learning-dynamics trends. These findings suggest that the distribution of post-training data, rather than algorithmic differences alone, plays a central role in forgetting, and highlight RFT as a promising ingredient for stable continual post-training.

多模态知识保留强化微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。