arXiv:2509.05542cs.LG2025-09

通过实例级权重优化,显著提升多模态推理模型的训练效果和稳定性。

DreamPRM-1.5: Unlocking the Potential of Each Instance for Multimodal Process Reward Model Training

  • 采用双层优化动态分配每个训练样本权重,实现细粒度数据重加权。
  • 在MMMU和R-Bench-V上分别达到84.6和31.3的准确率,接近理论上限。
  • 适合需要高精度多模态推理的系统开发者与研究者使用。

多模态过程奖励模型(PRMs)的训练面临两大挑战:训练集与测试集间的分布偏移,以及训练数据样本间质量不平衡。尽管已有方法(如DreamPRM)在领域层面进行重加权以对齐测试目标,但其性能仍与理想上限(pass@N)存在明显差距,即使在使用测试集数据进行“合理性检验”时亦然,表明存在元层级的参数不足问题。本文提出DreamPRM-1.5,一种实例级重加权框架,通过双层优化为每个训练样本动态分配自适应权重。为实现跨规模的实例重加权,设计了两种互补机制:Instance Table在小中规模数据上学习显式样本权重;Instance Net则是一个轻量神经网络,具备更强泛化能力,可扩展至大规模语料。结合一套实用且稳定的训练策略——上下层更新的时间尺度匹配、冷启动初始化及权重范围约束——有效防止训练发散。集成测试时缩放后,DreamPRM-1.5在MMMU验证集上达到84.6的准确率,在R-Bench-V上达31.3,并与领先骨干模型(如GPT-5-mini)结合,在公开多模态推理排行榜上取得第一名。大量实验(包括基准测试、基线对比与合理性检验)表明,该方法显著缩小与理想上限的差距,性能领先且训练稳定。

原文摘要 · Abstract (English)

Training multimodal process reward models (PRMs) is hard due to (i) distribution shift between training set and test set and (ii) quality imbalance across training data samples. While domain-level reweighting (e.g., DreamPRM) aligns training with test-time objectives, it leaves a clear gap to an oracle upper bound (pass@N), even under a "sanity check" that uses test set data to probe headroom -- pointing to meta-level under-parameterization. We introduce DreamPRM-1.5, an instance-level reweighting framework that assigns an adaptive weight to every training example via bi-level optimization. To realize instance reweighting across scales, we develop two complementary regimes: Instance Table, which learns explicit per-sample weights and excels on small/medium data, and Instance Net, a lightweight neural network that generalizes better and scales to large corpora. A practical, stable training recipe -- time-scale matching between upper/lower updates, cold-start initialization, and bounded-range weights -- prevents divergence. Integrated with test-time scaling, DreamPRM-1.5 attains 84.6 accuracy on the MMMU validation set, 31.3 accuracy on R-Bench-V and, when paired with a leading backbone (e.g., GPT-5-mini), achieves first-place results on public multimodal reasoning leaderboards. Moreover, extensive experiments, including benchmark evaluations, baseline comparisons, and a sanity check, demonstrate that DreamPRM-1.5 closes the gap toward the oracle, achieves leading performance, and trains stably.

多模态推理奖励建模数据重加权训练优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。