发现图文模型读图时不愿思考,导致性能下降,提出简单方法让模型恢复推理能力。
Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs
- 用自监督方式让模型模仿自己在文本模式下的推理过程来训练图像输入
- 图像模式准确率提升超50%,甚至超过文本模式表现
- 适合想提升视觉文本理解能力的模型开发者
多模态大语言模型可处理图像形式的文本,但性能常低于纯文本输入。我们系统评估7个MLLMs在7个基准上的5种输入模式(包括合成文本和arXiv、维基百科的真实文档图像),发现性能差距高度依赖字体、分辨率等渲染因素;真实文档图像的差距更小,表明部分差异源于评估偏差而非本质缺陷。对4000多个样本的扎根理论错误分析显示:图像输入会抑制模型推理意愿,输出长度缩短5–19倍,跳过分步计算。根本原因不是感知或知识获取失败,而是缺乏推理动力,尤其在多步任务中显著。通过轻量级在线策略自蒸馏方法——用模型自身文本模式推理轨迹配对图像输入进行微调,可有效弥合差距,使图像模式准确率提升超50%,达到或超越文本模式,并在未见基准上保持性能,无灾难性遗忘。结果为理解与解决模态鸿沟提供了系统性依据和实用路径。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) can process text presented as images, yet they often perform worse than when the same content is provided as textual tokens. We systematically diagnose this "modality gap" by evaluating seven MLLMs across seven benchmarks in five input modes, spanning both synthetically rendered text and realistic document images from arXiv PDFs to Wikipedia pages. We find that the gap is highly sensitive to rendering choices such as font and resolution, and that natural document images often exhibit much smaller gaps, suggesting the performance difference partly reflects evaluation artifacts rather than fundamental limitations. Through a grounded-theory error analysis of over 4,000 examples, we identify the primary cause: image input alone suppresses reasoning effort, with models producing 5--19x shorter outputs that skip step-by-step computation or reasoning. The reluctance to reason, not a failure of perception or knowledge retrieval, drives the performance gap, particularly on tasks requiring multi-step reasoning. We show that a simple, lightweight on-policy self-distillation method by fine-tuning models on their own text-mode reasoning traces paired with image inputs closes this gap, raising image-mode accuracy to match or exceed text-mode performance with over 50\% improvement, and the gains transfer to unseen benchmarks without catastrophic forgetting. Overall, our results and analyses provide a systematic understanding of the modality gap and suggest a practical path toward improving visual text understanding in multimodal language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。