arXiv:2606.17953cs.CV2026-06中稿 · IJCAI

模型先看对图像,后被文字误导,新方法能修复这一错误。

MLLMs Get It Right, Then Get It Wrong: Tracing and Correcting Late-Layer Textual Bias

论文配图:MLLMs Get It Right, Then Get It Wrong: Tracing and Correcting Late-Layer Textual Bias
图 1 · 摘自论文原文
  • 发现中间层视觉判断正确,但最终输出被文字干扰
  • 85%失败案例向文字偏移,89%成功案例向视觉回归
  • 无需训练,推理时恢复被压制的正确视觉判断

当视觉与文本矛盾时,多模态大语言模型(MLLMs)始终偏向文本,即使图像提供明确证据。本文揭示一个意外现象:模型在中间层常做出正确的视觉判断,但在最终输出阶段却转向文本,称为“晚层文本覆盖”。视觉信息虽被编码,却未传递至输出。更关键的是,预测变化方向可判断对错:85%的失败案例向文本偏移,89%的成功案例向视觉回归。据此提出无训练干预方法CALRD(冲突感知层参考解码),在推理时检测并恢复被抑制的置信视觉预测。五种不同架构的MLLM实验显示,在冲突基准上提升最高达9.4个百分点,同时保持标准性能不变,仅通过修正已有知识而非新增训练或外部信息。

原文摘要 · Abstract (English)

When vision contradicts text, multimodal large language models (MLLMs) consistently favor text, even when images provide clear evidence otherwise. This bias poses risks for applications requiring visual grounding, yet its cause remains unclear. In this paper, we uncover a surprising finding: models often get it right initially, forming correct vision-based predictions in their intermediate layers, before changing their minds and favoring text in the final output. We call this "late-layer textual override". The visual information is encoded, it simply does not survive to the output. More intriguingly, we find that how predictions change reveals whether they're correct: 85% of failures shift toward text, while 89% of successes shift toward vision. This directional signature enables a simple but powerful intervention: when we detect a confident visual prediction being suppressed, we restore it. We propose CALRD (Conflict-Aware Layer Reference Decoding), a training-free method that recovers overridden predictions at inference time. Experiments across five MLLMs of varying architectures demonstrate up to 9.4% absolute improvements on conflict benchmarks while largely preserving standard performance, without training or external knowledge. It recovers what the model already knew but failed to preserve.

多模态文本偏见推理修正

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。