通过反事实视觉对齐,让多模态模型不再依赖语言捷径,更准确地理解图像。
Staying VIGILant: Mitigating Visual Laziness via Counterfactual Visual Alignment in MLLMs

- 设计反事实盲态,惩罚模型在看不见图像时仍盲目自信的错误。
- 仅用25%数据达到顶尖方法全量数据性能,且减少幻觉现象。
- 无需标注框就能自发掌握空间位置关系,适合图像推理任务。
多模态大语言模型(MLLMs)虽具备强大文本推理能力,却常因视觉懒惰而产生与图像矛盾的幻觉:模型内部已编码正确视觉信息,但过度依赖语言先验生成答案。现有对齐方法主要优化文本输出奖励,导致模型倾向语言捷径,加剧视觉冲突。为此,我们提出视觉信息增益对齐(VIGIL),一种基于强化学习的后训练框架,将优化重点从数值奖励转向因果视觉锚定。VIGIL引入几何约束,显式最大化视觉输入与生成回答间的互信息,通过在遮蔽图文注意力后惩罚模型仍保持高置信度的情况,构建反事实盲态。大量实验表明,VIGIL在幻觉与推理基准上持续优于近期对齐方法,且不损害纯文本能力。该方法仅需25%偏好数据即可达到先进方法全量数据表现,并在无边界框监督下展现出涌现的空间定位能力。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) extend large language models (LLMs) with visual perception, enabling joint reasoning over images and text. Despite inheriting strong reasoning capabilities from LLMs, they remain prone to hallucinations that contradict their visual inputs. Mechanistic studies indicate that this weakness stems from visual laziness: MLLMs encode the correct visual evidence internally, but overly rely on strong language priors during response. Existing alignment methods, such as direct preference optimization, primarily optimize outcome-level rewards based on text. This introduces an optimization bias toward linguistic shortcuts, leading to responses that often contradict the visual evidence. To address this, we propose Visual Information Gain In aLignment (VIGIL), a reinforcement-learning (RL) post-training framework that shifts the focus from numerical reward fitting to causal visual grounding. VIGIL introduces a geometric constraint that explicitly maximizes the mutual information between the visual input and the generated response. We achieve this by penalizing "blind confidence" instances where the model remains improperly certain even when textual-visual attention is masked to create a counterfactual blind state. Extensive experiments show that VIGIL consistently outperforms recent alignment methods across hallucination and reasoning benchmarks without compromising text-only capabilities. Our approach matches the full-data performance of state-of-the-art methods using only 25% of the preference data and even demonstrates emergent spatial grounding capabilities without explicit bounding box supervision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。