让视觉模型更懂因果，减少胡说八道。
CFPO: Counterfactual Policy Optimization for Multimodal Reasoning
- 通过对比真实与屏蔽视觉线索的假想状态，强制模型理解视觉与语言的因果关系。
- 在多模态推理任务中，相比基线提升3.17%-6.25%，比先进方法还高1.32%-2.13%。
- 无需额外标注或奖励模型，可无缝接入主流强化学习算法，适合追求可靠推理的研究者。
大型视觉-语言模型在多模态推理中表现出色，但现有强化学习范式缺乏显式的反事实增强和因果学习机制，导致严重接地失败：模型倾向于忽视视觉证据而依赖语言先验，或在长链推理中产生幻觉漂移。为此，我们提出反事实策略优化（CFPO），一种强制视觉感知与文本推理间因果一致性的新框架。CFPO引入跨模态反事实增强机制，通过最大化模型预测与关键视觉线索被抑制时的反事实预测之间的差异，来正则化策略。该方法可无缝集成至标准算法如GRPO和DAPO，无需外部奖励模型或额外监督。大量实验表明，CFPO显著提升了推理保真度，在多个基准上相较标准强化学习基线提升3.17%-6.25%，较最先进的感知感知方法PAPO提升1.32%-2.13%。代码已开源。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities in multimodal reasoning. However, prevailing reinforcement learning (RL) paradigms lack explicit counterfactual enhancement and causal learning mechanisms. This fundamental deficiency results in severe grounding failures, manifesting as a tendency to ignore visual evidence in favor of language priors or exhibiting hallucination drift during long chain-of-thought reasoning. To address this root cause, we propose CounterFactual Policy Optimization (CFPO), a novel framework that enforces causal consistency between visual perception and textual reasoning. CFPO introduces a cross-modal counterfactual enhancement mechanism, which regularizes the policy by maximizing the discrepancy between the model's predictions and those from a counterfactual state where critical visual cues are suppressed. This approach seamlessly integrates with standard algorithms like GRPO and DAPO without requiring external reward models or additional supervision. Extensive experiments demonstrate that CFPO significantly improves reasoning fidelity, achieving consistent gains of 3.17%-6.25% over standard RL baselines and 1.32%-2.13% over the state-of-the-art perception-aware method (PAPO). Code is available at https://github.com/Raven-July/CFPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。