arXiv:2505.13427cs.AIcs.CV2025-05被引 28

用自动标注的步骤级监督,让多模态模型解数学题更逻辑严谨。

MM-PRM: Enhancing Multimodal Mathematical Reasoning with Scalable Step-Level Supervision

  • 通过蒙特卡洛树搜索自动生成超70万条推理步骤标注。
  • 在多个数学基准上,解题正确率显著提升,跨域泛化能力强。
  • 适合研究多模态推理、模型可解释性及自动化标注方向的人参考。

尽管多模态大语言模型在视觉-语言理解方面取得显著进展,但在复杂多步推理任务中仍表现不佳,常产生逻辑不一致或部分正确的答案。其核心问题在于缺乏对中间推理步骤的细粒度监督。为此,我们提出MM-PRM,一种在完全自动化、可扩展框架下训练的过程奖励模型。首先构建了基于多样化数学推理数据训练的强模型MM-Policy;随后构建了包含10,000个带可验证答案的多模态数学问题的数据集MM-K12作为种子数据。利用基于蒙特卡洛树搜索(MCTS)的流水线,无需人工标注即生成超过70万条步骤级标注。该奖励模型用于Best-of-N推理中的候选路径评分,在域内(MM-K12测试集)和域外(OlympiadBench、MathVista等)基准上均实现显著提升。进一步分析证实,软标签、较小学习率和路径多样性有助于优化奖励模型性能。结果表明,过程监督是提升多模态推理系统逻辑鲁棒性的有效手段。代码与数据已开源。

原文摘要 · Abstract (English)

While Multimodal Large Language Models (MLLMs) have achieved impressive progress in vision-language understanding, they still struggle with complex multi-step reasoning, often producing logically inconsistent or partially correct solutions. A key limitation lies in the lack of fine-grained supervision over intermediate reasoning steps. To address this, we propose MM-PRM, a process reward model trained within a fully automated, scalable framework. We first build MM-Policy, a strong multimodal model trained on diverse mathematical reasoning data. Then, we construct MM-K12, a curated dataset of 10,000 multimodal math problems with verifiable answers, which serves as seed data. Leveraging a Monte Carlo Tree Search (MCTS)-based pipeline, we generate over 700k step-level annotations without human labeling. The resulting PRM is used to score candidate reasoning paths in the Best-of-N inference setup and achieves significant improvements across both in-domain (MM-K12 test set) and out-of-domain (OlympiadBench, MathVista, etc.) benchmarks. Further analysis confirms the effectiveness of soft labels, smaller learning rates, and path diversity in optimizing PRM performance. MM-PRM demonstrates that process supervision is a powerful tool for enhancing the logical robustness of multimodal reasoning systems. We release all our codes and data at https://github.com/ModalMinds/MM-PRM.

多模态推理数学推理奖励模型自动化标注

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。