arXiv:2511.18437cs.CV2025-11被引 18

让视觉推理有据可查,防止模型胡说八道。

Perceptual-Evidence Anchored Reinforced Learning for Multimodal Reasoning

  • 用可验证的视觉问题清单检验模型看图能力
  • 在数学推理任务上比基线提升9.7%,比GRPO高6.6%
  • 适合需要严谨多模态推理的场景,如医疗、金融

基于可验证奖励的强化学习(RLVR)显著提升了大语言模型的推理能力,并正被应用于视觉语言模型(VLMs)。然而,现有的VLM-RLE仅验证最终文本输出,忽略了视觉感知这一基础步骤,导致视觉幻觉和奖励作弊。为此,我们提出PEARL(感知证据锚定强化学习),一种双分支、感知-推理协同的框架,通过显式锚定推理到已验证的视觉证据来增强多模态推理。针对每个问答实例,PEARL首先生成一组可验证的感知子问题构成检查清单,以评估模型对关键视觉证据的理解。训练时,通过该清单的辅助采样获得感知奖励,既直接强化感知能力,又作为推理的可信门控:若感知通过,则策略更新偏向证据锚定推理;否则中止流程,防止基于错误前提的推理。PEARL可无缝集成于GRPO和DAPO等主流强化学习方法。全面实验表明,PEARL在多模态推理基准上取得显著提升,在MathVerse上相比基线提升9.7%,相比GRPO提升6.6%。

原文摘要 · Abstract (English)

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capabilities of Large Language Models (LLMs) and is now being applied to Vision-Language Models (VLMs). However, vanilla RLVR for VLMs verifies only the final textual output, critically neglecting the foundational step of visual perception. This oversight leads to visual hallucinations and reward hacking, as reasoning built upon flawed perception is inherently unreliable. To address this, we propose PEARL (Perceptual-Evidence Anchored Reinforced Learning), a dual-branch, perception-reasoning synergistic that strengthens multimodal reasoning by explicitly anchoring it to verified visual evidence. For each reasoning-oriented QA instance, PEARL first derive a perception checklist -- a set of perception-oriented sub-questions with verifiable answers that probe the model's understanding of key visual evidence. During training, auxiliary rollouts on this checklist yield a perceptual reward that both directly reinforces the model's perception ability and acts as a fidelity gate for reasoning. If the model passes the perception check, its policy update is biased towards evidence-anchored reasoning. Otherwise, the process is halted to prevent reasoning from flawed premises. PEARL can be seamlessly integrated with popular RL methods like GRPO and DAPO. Comprehensive experiments show PEARL achieves substantial gains on multimodal reasoning benchmarks, e.g., a +9.7% improvement over the baseline and +6.6% over GRPO on MathVerse.

多模态推理强化学习视觉验证大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。