用强化学习激发推理能力,再用监督微调提升效果。
Metis-RISE: RL Incentivizes and SFT Enhances Multimodal Reasoning Model Learning
- 先用强化学习激活模型潜藏的推理能力,避免冷启动问题。
- 通过自蒸馏轨迹和专家知识增强,解决推理不一致与缺失问题。
- 72B模型在多模态推理榜单中排名第四,适合追求性能的开发者。
近期大型语言模型发展推动了先进推理范式的进步,并逐步融入多模态大模型(MLLMs)。现有方法存在局限:纯强化学习(RL)样本效率低且难以激活全新推理能力;而先监督微调(SFT)再RL的流程会限制探索并导致收敛不佳。本文提出Metis-RISE(RL激励、SFT增强),跳过初始SFT阶段,直接以强化学习(如组相对策略优化)启动,激励并激活模型潜在推理能力。随后的针对性SFT阶段解决两个关键问题:(1)对模型具备但应用不稳定的推理任务,采用来自RL模型自身的自蒸馏推理轨迹进行高效采样;(2)对完全无法处理的任务,引入专家增强的知识注入提示。该策略使7B与72B参数版本的MLLM在OpenCompass多模态推理排行榜上达到同类模型最优表现,72B版本位列整体第四。开源信息见项目页面。
原文摘要 · Abstract (English)
Recent advancements in large language models (LLMs) have witnessed a surge in the development of advanced reasoning paradigms, which are now being integrated into multimodal large language models (MLLMs). However, existing approaches often fall short: methods solely employing reinforcement learning (RL) can struggle with sample inefficiency and activating entirely absent reasoning capabilities, while conventional pipelines that initiate with a cold-start supervised fine-tuning (SFT) phase before RL may restrict the model's exploratory capacity and face suboptimal convergence. In this work, we introduce \textbf{Metis-RISE} (\textbf{R}L \textbf{I}ncentivizes and \textbf{S}FT \textbf{E}nhances) for multimodal reasoning model learning. Unlike conventional approaches, Metis-RISE distinctively omits an initial SFT stage, beginning instead with an RL phase (e.g., using a Group Relative Policy Optimization variant) to incentivize and activate the model's latent reasoning capacity. Subsequently, the targeted SFT stage addresses two key challenges identified during RL: (1) \textit{inefficient trajectory sampling} for tasks where the model possesses but inconsistently applies correct reasoning, which we tackle using self-distilled reasoning trajectories from the RL model itself; and (2) \textit{fundamental capability absence}, which we address by injecting expert-augmented knowledge for prompts where the model entirely fails. This strategic application of RL for incentivization followed by SFT for enhancement forms the core of Metis-RISE, leading to two versions of our MLLMs (7B and 72B parameters). Evaluations on the OpenCompass Multimodal Reasoning Leaderboard demonstrate that both models achieve state-of-the-art performance among similar-sized models, with the 72B version ranking fourth overall. Please refer to our project page for open-source information.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。