20亿参数视觉模型经强化学习后首次实现类'顿悟'推理，准确率提升30%。
R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Model
- 直接在无微调模型上用强化学习训练视觉推理能力
- 在CVBench上达到59.47%准确率，比基线高30%
- 揭示指令模型强化学习易导致简单推理路径
近期DeepSeek R1展示了基于规则激励的强化学习如何促使大语言模型自主发展复杂推理能力，表现为‘顿悟时刻’——模型在训练中展现出自我反思与响应长度增加。然而，将此成功扩展至多模态推理时，常无法复现这些关键特征。本报告首次在仅使用非微调的20亿参数模型上成功复现了多模态推理中的涌现特性。以Qwen2-VL-2B为基础，直接在SAT数据集上应用强化学习，模型在CVBench上达到59.47%准确率，较基线提升约30%，且超过微调设置约2%。此外，我们分享了尝试在指令模型上实现类R1推理的失败经历与洞察，旨在揭示挑战所在。关键观察包括：(1) 在指令模型上使用强化学习往往导致平凡的推理轨迹；(2) 简单的长度奖励无法有效激发推理能力。项目代码已公开于https://github.com/turningpoint-ai/VisualThinker-R1-Zero。
原文摘要 · Abstract (English)
Recently DeepSeek R1 demonstrated how reinforcement learning with simple rule-based incentives can enable autonomous development of complex reasoning in large language models, characterized by the "aha moment", in which the model manifest self-reflection and increased response length during training. However, attempts to extend this success to multimodal reasoning often failed to reproduce these key characteristics. In this report, we present the first successful replication of these emergent characteristics for multimodal reasoning on only a non-SFT 2B model. Starting with Qwen2-VL-2B and applying reinforcement learning directly on the SAT dataset, our model achieves 59.47% accuracy on CVBench, outperforming the base model by approximately ~30% and exceeding both SFT setting by ~2%. In addition, we share our failed attempts and insights in attempting to achieve R1-like reasoning using RL with instruct models. aiming to shed light on the challenges involved. Our key observations include: (1) applying RL on instruct model often results in trivial reasoning trajectories, and (2) naive length reward are ineffective in eliciting reasoning capabilities. The project code is available at https://github.com/turningpoint-ai/VisualThinker-R1-Zero
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。