SFT会诱导模型模仿伪推理,反而阻碍后续RL学习真实推理能力。
SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models
- 用新数据集验证SFT导致模型陷入机械模仿的僵化推理模式
- 基于GRPO与混合奖励机制的RL方法使推理更自然,准确率领先1.8%
- 适合研究视觉语言模型推理能力提升的学者和开发者参考
本文重新审视主流的监督微调(SFT)后强化学习(RL)训练范式,发现SFT会因模仿专家模型而引入‘伪推理路径’,这些路径看似合理却包含冗长、犹豫、低信息量且错误的步骤。为此,我们构建了多模态推理数据集VLAA-Thinking,包含高质量的分步视觉推理轨迹,并用于对比SFT、RL及其组合的效果。实验表明,尽管SFT有助于学习推理格式,但会使模型陷入机械模仿的僵化模式,阻碍进一步优化。相比之下,基于组相对策略优化(GRPO)并引入感知与认知信号混合奖励的新方法,能激发更真实、自适应的推理行为。基于Qwen2.5VL 3B的模型VLAA-Thinker在4B规模的Open LMM Reasoning Leaderboard上取得最高精度,超越此前最优结果1.8%。本研究为发展具备真实推理能力的视觉语言模型提供了关键洞见。
原文摘要 · Abstract (English)
This work revisits the dominant supervised fine-tuning (SFT) then reinforcement learning (RL) paradigm for training Large Vision-Language Models (LVLMs), and reveals a key finding: SFT can significantly undermine subsequent RL by inducing ``pseudo reasoning paths'' imitated from expert models. While these paths may resemble the native reasoning paths of RL models, they often involve prolonged, hesitant, less informative steps, and incorrect reasoning. To systematically study this effect, we introduce VLAA-Thinking, a new multimodal dataset designed to support reasoning in LVLMs. Constructed via a six-step pipeline involving captioning, reasoning distillation, answer rewrite and verification, VLAA-Thinking comprises high-quality, step-by-step visual reasoning traces for SFT, along with a more challenging RL split from the same data source. Using this dataset, we conduct extensive experiments comparing SFT, RL and their combinations. Results show that while SFT helps models learn reasoning formats, it often locks aligned models into imitative, rigid reasoning modes that impede further learning. In contrast, building on the Group Relative Policy Optimization (GRPO) with a novel mixed reward module integrating both perception and cognition signals, our RL approach fosters more genuine, adaptive reasoning behavior. Notably, our model VLAA-Thinker, based on Qwen2.5VL 3B, achieves top-1 performance on Open LMM Reasoning Leaderboard (https://huggingface.co/spaces/opencompass/Open_LMM_Reasoning_Leaderboard) among 4B scale LVLMs, surpassing the previous state-of-the-art by 1.8%. We hope our findings provide valuable insights in developing reasoning-capable LVLMs and can inform future research in this area.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。