arXiv:2501.17161cs.AIcs.CV2025-01ICML被引 703

强化学习比监督微调更擅长泛化,尤其在视觉和文本规则任务中。

SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training

  • 用奖励驱动的强化学习提升模型跨域泛化能力
  • 强化学习使模型在新规则下表现更好,而微调易陷入记忆陷阱
  • 适合关注模型泛化与可迁移性的研究人员

监督微调(SFT)和强化学习(RL)是基础模型后训练的常用方法,但其对模型泛化能力的影响尚不明确。本文通过对比 SFT 与 RL 在文本规则变体和视觉变体上的表现,引入 GeneralPoints(一个算术推理卡牌游戏)和 V-IRL(一个真实世界导航环境),评估模型在未见变体上的泛化能力。结果表明,基于结果奖励的强化学习能有效泛化至文本与视觉领域的新规则;而监督微调则倾向于记忆训练数据,难以应对分布外场景。进一步分析显示,强化学习提升了模型底层视觉识别能力,增强其在视觉任务中的泛化性。尽管强化学习具备更强泛化能力,但监督微调仍对稳定输出格式至关重要,为后续强化学习提供必要基础。研究证实了强化学习在复杂多模态任务中获取可迁移知识的能力。

原文摘要 · Abstract (English)

Supervised fine-tuning (SFT) and reinforcement learning (RL) are widely used post-training techniques for foundation models. However, their roles in enhancing model generalization capabilities remain unclear. This paper studies the difference between SFT and RL on generalization and memorization, focusing on text-based rule variants and visual variants. We introduce GeneralPoints, an arithmetic reasoning card game, and adopt V-IRL, a real-world navigation environment, to assess how models trained with SFT and RL generalize to unseen variants in both textual and visual domains. We show that RL, especially when trained with an outcome-based reward, generalizes across both rule-based textual and visual variants. SFT, in contrast, tends to memorize training data and struggles to generalize out-of-distribution scenarios. Further analysis reveals that RL improves the model's underlying visual recognition capabilities, contributing to its enhanced generalization in the visual domain. Despite RL's superior generalization, we show that SFT remains essential for effective RL training; SFT stabilizes the model's output format, enabling subsequent RL to achieve its performance gains. These findings demonstrates the capability of RL for acquiring generalizable knowledge in complex, multi-modal tasks.

强化学习模型泛化多模态微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。