让多模态大模型通过逐步奖励自我提升推理能力
R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization
- 采用分步组相对策略优化，动态奖励每一步推理
- 在8个基准上实现领先性能，显著优于模仿式训练
- 适合需要严谨逻辑推理的科研与工程场景
现有研究通常通过高质量链式思维数据的监督微调来提升多模态大语言模型（MLLMs）的推理能力，但往往导致模型仅机械模仿成功路径而无法理解错误路径。本文提出一种新型在线强化学习框架——分步组相对策略优化（StepGRPO），使MLLMs能通过简单、高效且密集的分步奖励实现自我改进。StepGRPO引入两种基于规则的推理奖励：分步推理准确度奖励（StepRAR）通过软关键步骤匹配技术奖励包含必要中间步骤的推理路径；分步推理有效性奖励（StepRVR）通过推理完整性和逻辑一致性评估策略奖励结构合理、逻辑连贯的推理过程。基于该框架，我们构建了具备出色分步推理能力的R1-VL系列模型。在8个基准上的大量实验表明，本方法显著优于现有方法。
原文摘要 · Abstract (English)
Recent studies generally enhance MLLMs' reasoning capabilities via supervised fine-tuning on high-quality chain-of-thought reasoning data, which often leads models to merely imitate successful reasoning paths without understanding what the wrong reasoning paths are. In this work, we aim to enhance the MLLMs' reasoning ability beyond passively imitating positive reasoning paths. To this end, we design Step-wise Group Relative Policy Optimization (StepGRPO), a new online reinforcement learning framework that enables MLLMs to self-improve reasoning ability via simple, effective and dense step-wise rewarding. Specifically, StepGRPO introduces two novel rule-based reasoning rewards: Step-wise Reasoning Accuracy Reward (StepRAR) and Step-wise Reasoning Validity Reward (StepRVR). StepRAR rewards the reasoning paths that contain necessary intermediate reasoning steps via a soft key-step matching technique, while StepRAR rewards reasoning paths that follow a well-structured and logically consistent reasoning process through a reasoning completeness and logic evaluation strategy. With the proposed StepGRPO, we introduce R1-VL, a series of MLLMs with outstanding capabilities in step-by-step reasoning. Extensive experiments over 8 benchmarks demonstrate the superiority of our methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。