让多模态模型推理更准:通过图像描述约束,减少视觉误解导致的错误。
Perception-Consistency Multimodal Large Language Models Reasoning via Caption-Regularized Policy Optimization
- 用图像描述约束模型输出,确保推理基于准确的视觉理解。
- 数学任务准确率提升6.0%,通用推理提升2.4%。
- 适合关注多模态推理鲁棒性的研究者和开发者。
尽管多模态大语言模型在融合视觉感知与符号推理的任务中表现优异,但其性能常受制于感知错误在推理链中的传播。现有强化学习微调方法虽提升了推理能力,却未能解决视觉定位与后续推理之间的根本错位问题。为此,我们提出一种名为【Caption-Regularized Policy Optimization (CapPO)】的新颖强化学习框架,通过在策略优化中显式强制感知一致性。该框架包含两项核心机制:(1) 基于图像描述的一致性正则化,最小化原始图像与对应描述条件下的响应差异,从而将推理锚定在语义忠实的视觉内容上;(2) 基于KL权重的优势估计方案,自适应调整强化信号,增强感知一致的推理路径,抑制虚假关联。在五个数学相关与五个通用推理基准上的大量实验表明,CapPO实现了有竞争力的表现,相较于基础Qwen2.5-VL-7B模型,在数学任务上准确率提升+6.0%,通用推理任务提升+2.4%。消融实验进一步验证了各组件的有效性,错误分析显示,相比基线模型,CapPO显著减少了感知相关错误。总体而言,CapPO提供了一个简单而有效的多模态推理改进框架。
原文摘要 · Abstract (English)
While multimodal large language models excel at tasks that integrate visual perception with symbolic reasoning, their performance is often undermined by a critical vulnerability: perception-induced errors that propagate through the reasoning chain. Current reinforcement learning (RL) fine-tuning methods, while enhancing reasoning abilities, largely fail to address the underlying misalignment between visual grounding and the subsequent reasoning process. To address this challenge, we propose \textbf{Caption-Regularized Policy Optimization (CapPO)}, a novel RL framework that explicitly enforces perceptual consistency during policy optimization. CapPO integrates two key mechanisms: (1) a caption-based consistency regularization, which minimizes the divergence between responses conditioned on raw images and those conditioned on captions, thereby anchoring reasoning to semantically faithful visual content; and (2) a KL-weighted advantage estimation scheme, which adaptively scales reinforcement signals to strengthen perceptually consistent trajectories while suppressing spurious correlations. Extensive experiments on five math-focused and five general reasoning benchmarks demonstrate that CapPO achieves competitive performance, yielding gains of +6.0% accuracy on math-related tasks and +2.4% on general reasoning tasks over the base Qwen2.5-VL-7B model. Moreover, ablation studies further confirm the effectiveness of each component, while error analysis reveals that CapPO significantly reduces perception-related mistakes compared with baselines. Overall, CapPO provides a simple yet effective framework for improving multimodal reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。