RL比SFT泛化更好,因为会自动筛选中等难度数据。
Why Does RL Generalize Better Than SFT? A Data-Centric Perspective on VLM Post-Training
- RL隐式筛选中等难度样本,提升泛化能力
- 用难样本训练SFT模型会显著降低泛化性能
- 新方法DC-SFT显式选中等难度数据,效果超越RL
大型视觉语言模型(VLM)后训练中存在明显泛化差距:通过强化学习(RL)微调的模型在分布外(OOD)任务上表现优于监督微调(SFT)。本文从数据视角解释该现象,认为RL的优势源于其隐式数据过滤机制,能优先选择中等难度样本。为验证此假设,我们系统评估了在不同难度数据集上训练的SFT模型的OOD泛化能力。结果表明,使用困难样本训练会严重损害泛化性能。受此启发,我们提出难度筛选的SFT(DC-SFT),显式根据样本难度筛选训练集。实验显示,DC-SFT不仅显著提升标准SFT的泛化能力,甚至超过基于RL的训练,且具备更高稳定性和计算效率。本工作揭示了VLM中分布外泛化差距的数据本质,并提供更高效实现鲁棒泛化的路径。代码已开源于https://github.com/byyx666/DC-SFT。
原文摘要 · Abstract (English)
The adaptation of large-scale Vision-Language Models (VLMs) through post-training reveals a pronounced generalization gap: models fine-tuned with Reinforcement Learning (RL) consistently achieve superior out-of-distribution (OOD) performance compared to those trained with Supervised Fine-Tuning (SFT). This paper posits a data-centric explanation for this phenomenon, contending that RL's generalization advantage arises from an implicit data filtering mechanism that inherently prioritizes medium-difficulty training samples. To test this hypothesis, we systematically evaluate the OOD generalization of SFT models across training datasets of varying difficulty levels. Our results confirm that data difficulty is a critical factor, revealing that training on hard samples significantly degrades OOD performance. Motivated by this finding, we introduce Difficulty-Curated SFT (DC-SFT), a straightforward method that explicitly filters the training set based on sample difficulty. Experiments show that DC-SFT not only substantially enhances OOD generalization over standard SFT, but also surpasses the performance of RL-based training, all while providing greater stability and computational efficiency. This work offers a data-centric account of the OOD generalization gap in VLMs and establishes a more efficient pathway to achieving robust generalization. Code is available at https://github.com/byyx666/DC-SFT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。