通过滚动采样生成步骤奖励,让多模态模型推理更连贯可靠。
ProcessThinker: Enhancing Multi-modal Large Language Models Reasoning via Rollout-based Process Reward

- 用滚动采样估算每步的成功率,实现无需标注的步骤级奖励
- 在4个视频评测上,显著提升基线模型的多步推理准确率
- 适合需要逻辑严谨性的视觉问答任务,如数学推理与长视频理解
视觉问答越来越依赖多步推理。现有基于可验证奖励的强化学习(RLVR)和组相对策略优化(GRPO)虽能提升多模态推理能力,但大多依赖稀疏的最终结果奖励,难以区分错误是源于早期无效路径还是后期小失误。常见方法是训练过程奖励模型(PRM)以获得步骤级监督,但这需大规模高质量思维链标注且增加训练成本。我们提出ProcessThinker,一种无需显式训练PRM的实用后训练流程。该方法先将推理轨迹重写为带步骤标签的格式,进行冷启动监督微调,再结合标准格式奖励与基于滚动采样的过程奖励执行GRPO。具体地,对每个中间步骤,采样多个后续延续,并以最终答案验证的成功率作为该步骤奖励。此设计提供密集信用分配,鼓励能稳定导向正确结论的推理步骤,有效缓解多步推理中的不一致或自相矛盾问题。在四个挑战性视频基准(Video-MMMU、MMVU、VideoMathQA、LongVideoBench)上,ProcessThinker持续优于基线模型Qwen3-VL-8B-Instruct。
原文摘要 · Abstract (English)
Visual question answering increasingly requires multi-step reasoning. Recent post-training with reinforcement learning under verifiable rewards (RLVR) and Group Relative Policy Optimization (GRPO) can improve multimodal reasoning, but most approaches rely on sparse outcome-only rewards. As a result, they struggle to tell whether an incorrect answer comes from a small mistake late in the reasoning or from an unhelpful trajectory from the start. A common solution is to train a process reward model (PRM) for step-level supervision, but this typically requires large-scale high-quality chain-of-thought annotations and additional training cost. We propose ProcessThinker, a practical post-training pipeline that provides step-level process rewards without training an explicit PRM. ProcessThinker first rewrites reasoning traces into a step-tagged format for cold-start supervised fine-tuning, then applies GRPO with a standard format reward and our rollout-based process reward. Concretely, for each intermediate step, we sample multiple continuations from that step and use the empirical success rate (final-answer verification) as the step reward. This gives dense credit assignment and encourages reasoning steps that more reliably support a correct conclusion, helping reduce inconsistent or self-contradictory progress across steps -- a key issue in logical reasoning. Across four challenging video benchmarks (Video-MMMU, MMVU, VideoMathQA, and LongVideoBench), ProcessThinker consistently improves over the baseline model Qwen3-VL-8B-Instruct
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。