arXiv:2506.04207cs.LGcs.AI2025-06被引 51

通过分阶段训练提升多模态模型推理能力,效果超越多数开源7B模型。

Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learning

论文配图:Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learning
图 1 · 摘自论文原文
  • 先用精选文本数据初始化,再分阶段进行多模态与纯文本强化学习。
  • 在MathVerse等6个基准上达到开源7B模型新高,包括AIME2024/2025挑战赛。
  • 揭示冷启动与梯度停滞问题,适合想改进多模态推理的开发者参考。

受Deepseek-R1在复杂文本任务中出色推理能力的启发,许多研究尝试直接对多模态大语言模型(MLLMs)应用强化学习(RL)以激发类似能力,但效果仍不理想。本文深入分析当前训练流程,发现三个关键现象:1)有效的冷启动初始化对提升MLLM推理至关重要;令人惊讶的是,仅用精心挑选的文本数据初始化即可使模型性能超过多个近期多模态推理模型,甚至在进入多模态强化学习前已具备优势。2)标准GRPO应用于多模态强化学习时存在梯度停滞问题,导致训练不稳定且性能下降。3)在多模态强化学习之后进行后续的纯文本强化学习,能进一步增强多模态推理能力。该分阶段训练策略有效平衡了感知基础与认知推理的发展。基于以上发现并解决多模态强化学习问题,我们提出ReVisual-R1,在MathVerse、MathVision、WeMath、LogicVista、DynaMath以及具有挑战性的AIME2024和AIME2025等多个基准上,成为开源7B MLLMs的新标杆。

原文摘要 · Abstract (English)

Inspired by the remarkable reasoning capabilities of Deepseek-R1 in complex textual tasks, many works attempt to incentivize similar capabilities in Multimodal Large Language Models (MLLMs) by directly applying reinforcement learning (RL). However, they still struggle to activate complex reasoning. In this paper, rather than examining multimodal RL in isolation, we delve into current training pipelines and identify three crucial phenomena: 1) Effective cold start initialization is critical for enhancing MLLM reasoning. Intriguingly, we find that initializing with carefully selected text data alone can lead to performance surpassing many recent multimodal reasoning models, even before multimodal RL. 2) Standard GRPO applied to multimodal RL suffers from gradient stagnation, which degrades training stability and performance. 3) Subsequent text-only RL training, following the multimodal RL phase, further enhances multimodal reasoning. This staged training approach effectively balances perceptual grounding and cognitive reasoning development. By incorporating the above insights and addressing multimodal RL issues, we introduce ReVisual-R1, achieving a new state-of-the-art among open-source 7B MLLMs on challenging benchmarks including MathVerse, MathVision, WeMath, LogicVista, DynaMath, and challenging AIME2024 and AIME2025.

多模态强化学习推理增强模型训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。