arXiv:2605.16999cs.LG2026-05

让视觉语言模型在推理时更靠谱,避免盲目自信。

Ranking-Aware Calibration for Reliable Multimodal Reinforcement Learning

论文配图:Ranking-Aware Calibration for Reliable Multimodal Reinforcement Learning
图 1 · 摘自论文原文
  • 用对比信号训练模型区分好与坏的推理路径。
  • 在干扰输入下,模型自信度下降,错误率显著降低。
  • 无需额外标注,适合提升多模态强化学习的可靠性。

强化学习后训练显著提升了视觉-语言模型的推理准确率,但其策略仍存在严重校准问题。终端正确性奖励无法区分高自信错误与低自信错误,也无法将自信度与视觉证据质量关联,尤其在输入受损或模糊时,模型仍会高自信地给出错误答案。本文提出排名感知校准(RAC),一种训练阶段框架,利用组级强化学习已生成的比较信号,无须额外标注。排名感知组损失要求更好轨迹的置信度高于较差轨迹;清洁-损坏成对损失则使置信度随视觉证据退化而减弱。该方法促使策略区分正确与错误推理路径,提升任务准确率。在Qwen2.5-VL和InternVL-3.5基础上,在六个多模态推理基准上测试,结果表明:排名感知损失显著提升准确率,成对损坏损失有效降低退化输入下的校准误差,二者结合在所有测试模型中实现最优校准,多数场景下提升准确率。

原文摘要 · Abstract (English)

Reinforcement learning post-training has substantially improved the reasoning accuracy of vision-language models, yet the resulting policies remain poorly calibrated. Terminal correctness rewards provide no gradient that penalizes confident errors more than uncertain ones and no signal that ties confidence to the quality of visual evidence, a gap that becomes especially severe under corrupted or ambiguous inputs where models continue to report high confidence on incorrect answers. We introduce Ranking-Aware Calibration (RAC), a training-time framework that supervises confidence using two comparison signals that group-based RL already produces at no additional labeling cost. The ranking-aware group loss enforces that a better rollout receives higher confidence than a worse one within the same prompt. The clean--corrupted pairwise loss enforces that confidence attenuates as visual evidence degrades. Because the ranking signal forces the policy to distinguish between correct and incorrect reasoning paths, it also reinforces task accuracy beyond what correctness rewards alone produce. Both losses require no external confidence annotations and integrate naturally with group-based RL post-training. We instantiate RAC on Qwen2.5-VL and InternVL-3.5 backbones and evaluate on six multimodal reasoning benchmarks under clean and corrupted inputs. Empirical results show that the ranking-aware loss substantially improves task accuracy by teaching the policy to discriminate between better and worse reasoning, while the pairwise corruption loss reduces calibration error under degraded inputs. Their combination achieves the best calibration across all tested backbones while improving accuracy in the majority of settings.

多模态强化学习校准视觉语言

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。