用强化学习提升多模态大模型的推理能力，效果接近顶尖模型。
Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models
- 构建无人工标注的20万条多模态思维链数据，用于模型冷启动。
- 通过渐进式抑制训练，在1万条数学数据上实现73.5%准确率。
- 适合研究多模态推理、强化学习与大模型对齐的学者使用。
DeepSeek-R1-Zero 已证明仅通过强化学习（RL）即可在语言模型中催生推理能力。受此启发，我们探索如何利用 RL 提升多模态大语言模型（MLLM）的推理能力。然而，直接使用 RL 难以激活如提问和反思等复杂推理行为，因缺乏高质量的多模态推理数据。为此，我们提出 Vision-R1 模型以增强多模态推理能力。首先，借助现有 MLLM 与 DeepSeek-R1，通过模态对齐与数据筛选，构建无需人工标注的 20 万条多模态思维链数据集（Vision-R1-cold），作为 Vision-R1 的冷启动初始化数据。为缓解冷启动后过度思考带来的优化挑战，我们提出渐进式思维抑制训练（PTST），并采用基于硬格式结果奖励函数的组相对策略优化（GRPO），在 1 万条多模态数学数据上逐步优化模型的正确复杂推理能力。大量实验表明，该模型在多个多模态数学推理基准上平均提升约 6%。Vision-R1-7B 在广泛使用的 MathVista 基准上达到 73.5% 准确率，仅比领先模型 OpenAI O1 低 0.4%。扩大多模态数学数据规模后，Vision-R1-32B 与 Vision-R1-72B 分别达到 76.4% 和 78.2% 的得分。相关数据集与代码将开源于：https://github.com/Osilly/Vision-R1。
原文摘要 · Abstract (English)
DeepSeek-R1-Zero has successfully demonstrated the emergence of reasoning capabilities in LLMs purely through Reinforcement Learning (RL). Inspired by this breakthrough, we explore how RL can be utilized to enhance the reasoning capability of MLLMs. However, direct training with RL struggles to activate complex reasoning capabilities such as questioning and reflection in MLLMs, due to the absence of substantial high-quality multimodal reasoning data. To address this issue, we propose the reasoning MLLM, Vision-R1, to improve multimodal reasoning capability. Specifically, we first construct a high-quality multimodal CoT dataset without human annotations by leveraging an existing MLLM and DeepSeek-R1 through modality bridging and data filtering to obtain a 200K multimodal CoT dataset, Vision-R1-cold dataset. It serves as cold-start initialization data for Vision-R1. To mitigate the optimization challenges caused by overthinking after cold start, we propose Progressive Thinking Suppression Training (PTST) strategy and employ Group Relative Policy Optimization (GRPO) with the hard formatting result reward function to gradually refine the model's ability to learn correct and complex reasoning processes on a 10K multimodal math dataset. Comprehensive experiments show our model achieves an average improvement of $\sim$6% across various multimodal math reasoning benchmarks. Vision-R1-7B achieves a 73.5% accuracy on the widely used MathVista benchmark, which is only 0.4% lower than the leading reasoning model, OpenAI O1. Scaling up the amount of multimodal math data in the RL training, Vision-R1-32B and Vison-R1-72B achieves 76.4% and 78.2% MathVista benchmark scores, respectively. The datasets and code will be released in: https://github.com/Osilly/Vision-R1 .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。