通过双重不确定性引导,让模型更聪明地应对视觉模糊和决策不确定。
Dual-Uncertainty Guided Policy Learning for Multimodal Reasoning
- 用对称KL散度和策略熵分别量化感知与输出不确定性
- 在3B/7B/4B/8B模型上提升最多12.4%准确率
- 适合需要精准推理的多模态AI研究者
基于可验证奖励的强化学习(RLVR)提升了多模态大语言模型的推理能力。然而,现有方法通常将视觉输入视为确定性,忽视了视觉模态固有的感知模糊性。这导致无法区分模型的不确定性是源于复杂推理还是感知模糊,从而难以有针对性地分配探索或学习信号。为此,我们提出双不确定性引导策略学习方法DUPL,通过对称KL散度量化感知不确定性,用策略熵衡量输出不确定性,构建不确定性驱动的反馈循环,并引入动态分支优先机制,使策略优势聚焦于高感知或高决策模糊的状态,实现超越被动数据增强的有效定向探索。在涵盖数学与通用领域的多个多模态推理基准上评估,DUPL显著提升性能:在Qwen2.5-VL上准确率最高提升12.3%(3B)和7.9%(7B),在Qwen3-VL-Instruct上最高提升10.7%(4B)和12.4%(8B),持续优于GRPO;并可无缝适配其他算法(DAPO,平均+6.5%)与架构(LLaVA-OneVision-1.5,平均+4.7%)。结果表明,DUPL是一种有效且通用的多模态RLVR方法。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) has advanced reasoning capabilities in multimodal large language models. However, existing methods typically treat visual inputs as deterministic, overlooking the perceptual ambiguity inherent to the visual modality. Consequently, they fail to distinguish whether a model's uncertainty stems from complex reasoning or ambiguous perception, preventing the targeted allocation of exploration or learning signals. To address this gap, we introduce \textbf{DUPL}, a dual-uncertainty guided policy learning approach for multimodal RLVR that quantifies and leverages both perceptual uncertainty (via symmetric KL divergence) and output uncertainty (via policy entropy) to guide policy updates. By establishing an uncertainty-driven feedback loop and employing a dynamic branch prioritization mechanism, DUPL recalibrates the policy advantage to focus learning on states with high perceptual or decisional ambiguity, enabling effective targeted exploration beyond passive data augmentation. Evaluated on diverse multimodal reasoning benchmarks spanning mathematical and general domains, DUPL achieves solid gains. It improves Qwen2.5-VL accuracy by up to $\textbf{12.3%}$ (3B) and $\textbf{7.9%}$ (7B), and Qwen3-VL-Instruct by up to $\textbf{10.7%}$ (4B) and $\textbf{12.4%}$ (8B), consistently outperforming GRPO, while seamlessly generalizing to alternative algorithms (DAPO, $\textbf{+6.5%}$ avg) and architectures (LLaVA-OneVision-1.5, $\textbf{+4.7%}$ avg). These results demonstrate that DUPL is an effective and generalizable approach for multimodal RLVR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。