用共享推理路径提升多模态大模型的逻辑推理能力
R1-ShareVL: Incentivizing Reasoning Capability of Multimodal Large Language Models via Share-GRPO
- 通过扩展问题空间并共享推理轨迹增强探索
- 在六个基准上显著优于现有方法，提升推理准确率
- 适合研究多模态推理与强化学习融合的学者
本文旨在通过强化学习激励多模态大语言模型（MLLMs）的推理能力，并提出一种新方法Share-GRPO，以缓解强化学习中的稀疏奖励与优势消失问题。该方法首先通过数据变换技术扩展原始问题空间，随后引导MLLM在扩展后的问题空间中探索多样化推理路径，并在训练过程中跨问题共享已发现的推理轨迹。此外，Share-GRPO在优势计算中也共享奖励信息，实现对解法优势的分层估计，提升了相对优势评估的准确性，增强了策略训练的稳定性。在六个主流推理基准上的广泛评估表明，该方法性能显著领先。代码将公开于https://github.com/HJYao00/R1-ShareVL。
原文摘要 · Abstract (English)
In this work, we aim to incentivize the reasoning ability of Multimodal Large Language Models (MLLMs) via reinforcement learning (RL) and develop an effective approach that mitigates the sparse reward and advantage vanishing issues during RL. To this end, we propose Share-GRPO, a novel RL approach that tackle these issues by exploring and sharing diverse reasoning trajectories over expanded question space. Specifically, Share-GRPO first expands the question space for a given question via data transformation techniques, and then encourages MLLM to effectively explore diverse reasoning trajectories over the expanded question space and shares the discovered reasoning trajectories across the expanded questions during RL. In addition, Share-GRPO also shares reward information during advantage computation, which estimates solution advantages hierarchically across and within question variants, allowing more accurate estimation of relative advantages and improving the stability of policy training. Extensive evaluations over six widely-used reasoning benchmarks showcase the superior performance of our method. Code will be available at https://github.com/HJYao00/R1-ShareVL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。