arXiv:2509.21268cs.CV2025-09被引 38

提升多模态推理模型性能,通过增强数据采样与开放资源解决训练不稳问题。

MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resources

  • 提出方差感知采样策略,提升奖励方差以稳定强化学习优化
  • 构建160万条高质量长链思维数据与1.5万条强化学习问答对
  • 开源多尺度模型、代码与数据,支持可复现研究

大规模多模态推理模型虽进展迅速,但受限于缺乏开放的高质量长链思维(CoT)数据以及后训练阶段强化学习(RL)算法的不稳定性。主流的组相对策略优化(GRPO)在奖励方差较低时易出现梯度消失,削弱优化信号并影响收敛。本文提出方差感知采样(VAS),基于方差促进评分(VPS)结合结果方差与轨迹多样性,有效提升奖励方差,稳定策略优化。同时发布约160万条长链思维冷启动数据和约1.5万条强化学习问答对,确保数据质量、难度与多样性,并提供完整可复现的端到端训练代码库。此外,开源多个规模的多模态推理模型,建立社区标准化基准。数学推理基准测试验证了所提数据与VAS的有效性。消融实验与理论分析表明,奖励方差下界决定了期望策略梯度大小,而VAS是实现该保障的实用机制。代码、数据与检查点已公开于https://github.com/LengSicong/MMR1。

原文摘要 · Abstract (English)

Large multimodal reasoning models have achieved rapid progress, but their advancement is constrained by two major limitations: the absence of open, large-scale, high-quality long chain-of-thought (CoT) data, and the instability of reinforcement learning (RL) algorithms in post-training. Group Relative Policy Optimization (GRPO), the standard framework for RL fine-tuning, is prone to gradient vanishing when reward variance is low, which weakens optimization signals and impairs convergence. This work makes three contributions: (1) We propose Variance-Aware Sampling (VAS), a data selection strategy guided by Variance Promotion Score (VPS) that combines outcome variance and trajectory diversity to promote reward variance and stabilize policy optimization. (2) We release large-scale, carefully curated resources containing ~1.6M long CoT cold-start data and ~15k RL QA pairs, designed to ensure quality, difficulty, and diversity, along with a fully reproducible end-to-end training codebase. (3) We open-source a family of multimodal reasoning models in multiple scales, establishing standardized baselines for the community. Experiments across mathematical reasoning benchmarks demonstrate the effectiveness of both the curated data and the proposed VAS. Comprehensive ablation studies and analyses provide further insight into the contributions of each component. In addition, we theoretically establish that reward variance lower-bounds the expected policy gradient magnitude, with VAS serving as a practical mechanism to realize this guarantee. Our code, data, and checkpoints are available at https://github.com/LengSicong/MMR1.

多模态推理强化学习数据集可复现

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。