通过迭代SFT与强化学习,让小模型实现复杂视觉语言推理。
OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cycles
- 交替使用监督微调和强化学习,逐步激发模型推理能力。
- 7B模型在6个基准上提升显著,数学推理任务最高增3.8%。
- 适合研究多模态推理、模型自进化方向的学者参考。
我们提出OpenVLThinker,首个开源的大规模视觉语言模型(LVLM),能实现复杂的思维链推理,在挑战性视觉推理任务中表现优异。尽管纯文本推理模型(如Deepseek R1)在文本任务中表现良好,但通过监督微调(SFT)将其推理能力迁移到视觉语言模型时,常因视觉定位不准导致性能下降。而纯强化学习(RL)方法面临搜索空间过大问题,难以在小型模型(如7B LVLM)中催生反思行为。令人意外的是,交替进行SFT与RL仅数轮后便带来显著性能提升。分析显示,基础模型初始极少表现出推理行为,但SFT能有效激活其潜在能力并缩小RL搜索空间,加速推理能力发展;后续RL阶段进一步优化推理技能,生成更高质量的SFT数据以实现持续自我提升。OpenVLThinker-7B在六个需数学与通用推理的任务上持续领先,显著提升MathVista(+3.8%)、EMMA(+2.4%)和HallusionBench(+1.6%)。该工作揭示了SFT与RL协同对复杂推理任务的促进作用,并为在多模态场景中实现类似R1的推理能力提供早期证据。代码、模型与数据已公开于https://github.com/yihedeng9/OpenVLThinker。
原文摘要 · Abstract (English)
We introduce OpenVLThinker, one of the first open-source large vision-language models (LVLMs) to exhibit sophisticated chain-of-thought reasoning, achieving notable performance gains on challenging visual reasoning tasks. While text-based reasoning models (e.g., Deepseek R1) show promising results in text-only tasks, distilling their reasoning into LVLMs via supervised fine-tuning (SFT) often results in performance degradation due to imprecise visual grounding. Conversely, purely reinforcement learning (RL)-based methods face a large search space, hindering the emergence of reflective behaviors in smaller models (e.g., 7B LVLMs). Surprisingly, alternating between SFT and RL ultimately results in significant performance improvements after a few iterations. Our analysis reveals that the base model rarely exhibits reasoning behaviors initially, but SFT effectively surfaces these latent actions and narrows the RL search space, accelerating the development of reasoning capabilities. Each subsequent RL stage further refines the model's reasoning skills, producing higher-quality SFT data for continued self-improvement. OpenVLThinker-7B consistently advances performance across six benchmarks demanding mathematical and general reasoning, notably improving MathVista by 3.8%, EMMA by 2.4%, and HallusionBench by 1.6%. Beyond demonstrating the synergy between SFT and RL for complex reasoning tasks, our findings provide early evidence towards achieving R1-style reasoning in multimodal contexts. The code, model and data are held at https://github.com/yihedeng9/OpenVLThinker.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。