arXiv:2604.01840cs.AI2026-04被引 3

让视觉关键词获得更强学习信号,提升大模型多模态推理能力

Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models

  • 基于视觉依赖度动态分配奖励,聚焦关键视觉决策点
  • 在7个基准上平均提升18.7%,显著降低梯度方差
  • 适合需要精准视觉感知的多模态推理任务研究者

尽管可验证奖励的强化学习(RLVR)推动了大视觉语言模型(LVLMs)的推理能力,但现有框架存在根本性方法缺陷:对所有生成标记分配相同优势,从而稀释了多模态推理中至关重要的视觉基础步骤的学习信号。为此,我们提出「标记视觉依赖度」,通过视觉条件与仅文本预测分布之间的KL散度量化视觉输入的因果信息增益。结果显示该依赖度高度稀疏且语义关键。因此,我们引入感知基政策优化(PGPO),一种细粒度信用分配框架,可在标记层面动态重塑优势。通过阈值门控、质量守恒机制,PGPO主动增强视觉依赖标记的学习信号,同时抑制语言先验带来的梯度噪声。基于Qwen2.5-VL系列在七个挑战性多模态推理基准上的实验表明,PGPO平均提升18.7%。理论与实证分析均证实,PGPO有效降低梯度方差,防止训练崩溃,并作为强正则化器,促进稳健的感知基多模态推理。代码将发布于https://github.com/Yzk1114/PGPO。

原文摘要 · Abstract (English)

While Reinforcement Learning from Verifiable Rewards (RLVR) has advanced reasoning in Large Vision-Language Models (LVLMs), prevailing frameworks suffer from a foundational methodological flaw: by distributing identical advantages across all generated tokens, these methods inherently dilute the learning signals essential for optimizing the critical, visually-grounded steps of multimodal reasoning. To bridge this gap, we formulate \textit{Token Visual Dependency}, quantifying the causal information gain of visual inputs via the Kullback-Leibler (KL) divergence between visual-conditioned and text-only predictive distributions. Revealing that this dependency is highly sparse and semantically pivotal, we introduce Perception-Grounded Policy Optimization (PGPO), which is a novel fine-grained credit assignment framework that dynamically reshapes advantages at the token level. Through a threshold-gated, mass-conserving mechanism, PGPO actively amplifies learning signals for visually-dependent tokens while suppressing gradient noise from linguistic priors. Extensive experiments based on the Qwen2.5-VL series across seven challenging multimodal reasoning benchmarks demonstrate that PGPO boosts models by 18.7% on average. Both theoretical and empirical analyses confirm that PGPO effectively reduces gradient variance, prevents training collapse, and acts as a potent regularizer for robust, perception-grounded multimodal reasoning. Code will be released on https://github.com/Yzk1114/PGPO.

多模态推理强化学习视觉依赖

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。