通过一致性采样提升多模态大模型的强化学习推理准确率
Enhancing the Outcome Reward-based RL Training of MLLMs with Self-Consistency Sampling
- 对视觉输入微调并重复采样推理轨迹，用一致性评分筛选可靠路径
- 在6个基准上最高提升7.7个百分点，计算开销几乎不变
- 适用于多种模型，是解决奖励误导问题的通用方案
基于结果奖励的强化学习是优化多模态大语言模型逐步推理的常用方法。在多选题设置中，错误推理却猜对答案的轨迹会获得与真实推理相同的奖励，这一缺陷常被忽视。本文提出自一致性采样（SCS）来修正该问题：对每道题，(i) 引入小幅度视觉扰动，(ii) 重复截断并重采样初始推理轨迹；多个轨迹间的一致性生成可微分的一致性得分，在策略更新时降低不可靠轨迹权重。基于 Qwen2.5-VL-7B-Instruct，在 RLOO、GRPO 与 REINFORCE++ 系列方法中引入 SCS 后，六个多模态基准测试准确率最高提升 7.7 个百分点，计算开销几乎无增加。SCS 在 Qwen2.5-VL-3B-Instruct 和 InternVL3-8B 上也表现显著，为多模态大模型的结果奖励强化学习提供了一种简单通用的改进方案。
原文摘要 · Abstract (English)
Outcome-reward reinforcement learning (RL) is a common and increasingly significant way to refine the step-by-step reasoning of multimodal large language models (MLLMs). In the multiple-choice setting - a dominant format for multimodal reasoning benchmarks - the paradigm faces a significant yet often overlooked obstacle: unfaithful trajectories that guess the correct option after a faulty chain of thought receive the same reward as genuine reasoning, which is a flaw that cannot be ignored. We propose Self-Consistency Sampling (SCS) to correct this issue. For each question, SCS (i) introduces small visual perturbations and (ii) performs repeated truncation and resampling of an initial trajectory; agreement among the resulting trajectories yields a differentiable consistency score that down-weights unreliable traces during policy updates. Based on Qwen2.5-VL-7B-Instruct, plugging SCS into RLOO, GRPO, and REINFORCE++ series improves accuracy by up to 7.7 percentage points on six multimodal benchmarks with negligible extra computation. SCS also yields notable gains on both Qwen2.5-VL-3B-Instruct and InternVL3-8B, offering a simple, general remedy for outcome-reward RL in MLLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。