用结构化步骤奖励让多模态推理自动纠错,效率更高。
StructReward: Efficient Structured Process Rewards for Self-Correcting Multimodal Reasoning

- 通过轻量规则对齐推理步骤生成细粒度奖励
- 实验显示推理纠错能力提升,计算开销降低
- 适合想提升多模态推理自修正能力的研究者
基于可验证奖励的强化学习(RLVR)在提升多模态推理方面表现优异。但现有方法仅以最终答案正确性判断二值奖励,忽略了中间推理步骤的监督信息。过程奖励模型虽可提供更细粒度反馈,却通常依赖独立训练的验证器、昂贵的思维链标注或大语言模型在线评判。本文提出 StructReward,一种计算高效的框架,通过结构化步骤级奖励对齐提供密集强化信号。该框架将生成解视为一系列推理步骤,利用轻量级数值、符号和词法匹配规则与标注参考步骤对齐,生成密集过程奖励,并结合最终答案一致性和输出有效性奖励,通过门控组相对策略优化(GRPO)目标进行联合优化。此外,政策回放轨迹被回收用于响应对比和反思式自我修正,而非丢弃。同时,使用强语言模型将采样正确轨迹重写为反思导向的训练样本,进一步增强策略评估与优化推理的能力。由于奖励计算为在线执行,无需额外训练验证器或外部大模型评判,StructReward显著降低多模态强化学习的计算开销。实验表明,结构化过程监督与回放回收共同提供了一条高效实现自提升多模态推理的路径。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) has emerged as an effective approach for improving multimodal reasoning. However, most existing methods evaluate an entire response using a binary reward based only on final-answer correctness, thereby discarding the supervision available in intermediate reasoning steps. Process reward models offer finer-grained feedback, but they typically rely on separately trained verifiers, costly chain-of-thought annotations, or online judging by large language models (LLMs). In this work, we introduce StructReward, a compute-efficient framework that provides dense reinforcement signals through structured step-level reward alignment. StructReward represents each generated solution as a sequence of reasoning steps and aligns them with process-labeled reference steps using lightweight numerical, symbolic, and lexical matching rules. The aligned labels are aggregated into a dense process reward and combined with final-answer consistency and output-validity rewards through a gated Group Relative Policy Optimization (GRPO) objective. We further recycle policy rollouts into complementary supervision for response comparison and reflective self-correction, rather than discarding them after policy updates. Separately, we use a strong LLM to rewrite sampled correct trajectories into reflection-oriented training instances, further strengthening the policy's ability to evaluate and refine its reasoning. Since reward computation is performed online without an additional learned verifier or external LLM judge, StructReward substantially reduces the computational overhead of multimodal reinforcement learning. Experimental results show that structured process supervision and rollout recycling provide an efficient path toward self-improving multimodal reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。