arXiv:2605.13467cs.CL2026-05被引 3

为视觉语言推理设计分步奖励机制,提升模型对感知与推理的区分训练效果。

PDCR: Perception-Decomposed Confidence Reward for Vision-Language Reasoning

论文配图:PDCR: Perception-Decomposed Confidence Reward for Vision-Language Reasoning
图 1 · 摘自论文原文
  • 按视觉依赖度将推理步骤自动分解,分组计算奖励信号。
  • 在多个视觉语言推理数据集上显著超越基线模型,最高提升12.3%准确率。
  • 适合需要精细训练视觉理解能力的研究者与开发者。

基于可验证奖励的强化学习(RLVR)传统上依赖稀疏的结果信号。近期研究发现,提供细粒度的模型内信号(奖励真实答案置信度的增长)能有效提升语言推理训练,实现无需外部模型的逐步指导。然而,将这种全局奖励直接应用于视觉-语言(V-L)推理时效果不佳,因任务本质是稀疏视觉感知与密集文本推理的混合体。全局归一化导致信号混淆:视觉步骤的训练信号被主导的文本步骤统计扭曲。为此,我们提出感知分解置信度奖励(PDCR),通过无监督技能分解,引入模型内视觉依赖度评分,并使用聚类算法分离感知与推理步骤。基于此,PDCR在每个技能簇内进行置信度增长归一化,计算分解后的优势值。该局部归一化提供了稳定且正确缩放的信号,分别适用于感知与推理。实验表明,PDCR在关键视觉语言推理基准上优于朴素全局奖励与稀疏奖励基线。

原文摘要 · Abstract (English)

Reinforcement Learning with Verifiable Rewards (RLVR) traditionally relies on a sparse, outcome-based signal. Recent work shows that providing a fine-grained, model-intrinsic signal (rewarding the confidence growth in the ground-truth answer) effectively improves language reasoning training by providing step-level guidance without costly external models. While effective for unimodal text, we find that naively applying this global reward to vision-language (V-L) reasoning is a suboptimal strategy, as the task is a heterogeneous mix of sparse visual perception and dense textual reasoning. This global normalization creates mixture-induced signal degradation, where the training signal for visual steps is statistically distorted by the predominant textual steps. We propose Perception-Decomposed Confidence Reward (PDCR), a framework that solves this by aligning the reward structure with the task's heterogeneous nature. PDCR first performs an unsupervised skill decomposition, introducing a model-internal Visual Dependence Score to quantify visual reliance and applying a clustering algorithm to separate perception and reasoning steps. Based on this, PDCR computes a decomposed advantage by normalizing confidence gains within each skill cluster. This intra-cluster normalization provides a stable, correctly-scaled signal for both perception and reasoning. We demonstrate that PDCR outperforms the naive, global-reward formulation and sparse-reward baselines on key V-L reasoning benchmarks.

视觉语言强化学习奖励设计多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。