用互信息筛选视觉描述，提升多模态模型推理准确率
MIRL: Mutual Information-Guided Reinforcement Learning for Vision-Language Models
- 用视觉与描述的互信息预筛选高潜力轨迹，节省采样预算
- 仅用10个预采样+6个最优选择，达70.22%平均准确率
- 适合需要高效推理的复杂视觉语言任务研究者
视觉语言模型在复杂推理任务中常因视觉感知错误和幻觉导致答案不准确。基于可验证奖励的强化学习（RLVR）通过答案正确性信号优化策略，但现有方法存在两大局限：其一，大量采样预算浪费在早期视觉描述错误即注定失败的轨迹上；其二，稀疏奖励无法区分失败是源于视觉感知还是推理阶段。本文提出MIRL，一种解耦框架，利用生成描述与视觉输入间的互信息（MI）作为低成本预筛选信号。该机制实现智能预算分配，通过分叉策略聚焦高潜力轨迹，同时解耦训练提供独立的MI奖励用于优化视觉感知，解决奖励盲区问题。在六个视觉语言推理基准测试中，MIRL达到70.22%平均准确率，仅需10个预采样与前6个最优选择（相当于25%更少的完整轨迹），即可超越采样16条完整轨迹的性能。代码已公开。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) frequently suffer from visual perception errors and hallucinations that compromise answer accuracy in complex reasoning tasks. Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising solution by optimizing policies using answer correctness signals. Despite their effectiveness, prevailing RLVR methods face two critical limitations. First, much of the sampling budget is wasted on trajectories doomed to fail due to early visual description errors. Second, sparse rewards cannot distinguish whether failures stem from visual perception or reasoning stages. We introduce MIRL, a decoupled framework that addresses both limitations by leveraging mutual information (MI) between generated descriptions and visual inputs as a cheap pre-screening signal. This enables intelligent budget allocation toward high-potential trajectories via forking, while decoupled training provides independent MI-based rewards for visual perception optimization, resolving reward blindness. Experiments on six vision-language reasoning benchmarks demonstrate that MIRL achieves 70.22% average accuracy and successfully surpasses the performance of sampling 16 complete trajectories using only 10 pre-samples with top-6 selection (25% fewer complete trajectories). Our code is available at: https://anonymous.4open.science/r/mirl-main/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。