让视觉语言模型的推理更可靠,通过区分视觉提取和逻辑推理的 token 来精准优化。
Structured Role-Aware Policy Optimization for Multimodal Reasoning

- 按功能区分视觉提取与逻辑推理的 token,实现细粒度奖励分配。
- 在多个多模态推理基准上,证据关联性显著提升,正确率最高增 12.3%。
- 无需额外奖励模型,自蒸馏对比机制可直接提升推理可信度。
基于可验证奖励的强化学习(RLVR),尤其是组相对策略优化(GRPO),在提升大视觉语言模型(LVLMs)推理能力方面展现出强大潜力。然而,在多模态推理中,最终答案的奖励通常以序列级形式分配,无法区分不同 token 的功能角色,难以判断正确答案是否由任务相关的视觉证据支持。本文从角色感知的逐 token 信用分配视角重新审视多模态 RLVR,将结构化响应分解为用于提取视觉证据的感知 token 与基于该证据推导答案的推理 token。在此基础上,提出结构化角色感知策略优化(SRPO),在不改变原始奖励函数的前提下,将序列级 GRPO 优势重构为角色感知的逐 token 优势。具体地,SRPO 通过自蒸馏的在线策略对比:感知 token 根据其在原始与损坏视觉输入下的视觉依赖性被强调;推理 token 则根据其与生成感知的一致性被强调。这些角色特异性信号通过共享的轨迹级基线统一,生成正向 token 权重,调整相对更新幅度,同时保留原始 GRPO 奖励与优化方向,无需外部奖励模型或独立教师。在多个多样化的多模态推理基准上的实验表明,SRPO 显著提升了基于证据的推理性能,凸显了从均匀序列级信用转向角色感知优化对可靠多模态推理的重要性。
原文摘要 · Abstract (English)
Reinforcement learning from verifiable rewards (RLVR), especially with Group Relative Policy Optimization (GRPO), has shown strong potential for improving the reasoning capabilities of large vision-language models (LVLMs). However, in multimodal reasoning, final-answer rewards are typically assigned at the sequence level and do not distinguish the functional roles of different tokens, making it difficult to determine whether a correct answer is supported by task-relevant visual evidence. In this paper, we revisit multimodal RLVR from the perspective of role-aware token-level credit assignment, where structured responses are decomposed into perception tokens for extracting visual evidence and reasoning tokens for deriving answers from that evidence. Based on this perspective, we propose Structured Role-aware Policy Optimization (SRPO), which refines the sequence-level GRPO advantage into role-aware token-level advantages without changing the reward function. Specifically, SRPO assigns role-specific credit by using self-distilled on-policy contrasts: perception tokens are emphasized according to their visual dependency under original versus corrupted visual inputs, while reasoning tokens are emphasized according to their consistency with the generated perception. These role-specific signals are further unified through a shared trajectory-level baseline, yielding positive token weights that adjust relative update magnitudes while preserving the original GRPO reward and optimization direction, without requiring external reward models or separate teachers. Experiments across diverse multimodal reasoning benchmarks show that SRPO improves evidence-grounded reasoning, highlighting the importance of moving beyond uniform sequence-level credit toward role-aware optimization for reliable multimodal reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。