让大模型真正看懂图像中的文字,而非依赖提示词套路。
SimpleOCR: Rendering Visualized Questions to Teach MLLMs to Read
- 将问题直接渲染到图像上,强制模型必须读图而非猜题。
- 在新设定下性能下降12.7%,暴露模型“懒得看图”的弱点。
- 只需少量数据训练,就能显著提升模型读图能力,适合所有视觉语言模型。
尽管多模态大模型进展迅速,但其视觉定位机制的核心问题仍未解决:这些模型是否真正‘阅读’图像中的文字,还是仅依赖提示词中的参数捷径?本文通过引入可视化问题(VQ)设置,将文本查询直接渲染在图像上,强制模型进行视觉交互。在Qwen2.5-VL上的诊断实验显示,尽管具备强大OCR能力,模型在VQ设置下性能最高下降12.7%,暴露出深层的“模态惰性”。为此,我们提出SimpleOCR,一种即插即用的训练策略,通过随机风格化地将训练样本转换为VQ格式,有效消除基于文本的捷径,迫使模型激活并优化其视觉文本提取路径。实验证明,SimpleOCR无需修改架构即可带来显著提升:在四个代表性外部数据集上,相比基线模型提高5.4%,比基于原始图像的GRPO方法提升2.7%;且数据效率极高,仅需8.5K样本(为近期基于强化学习方法的30分之一)即达到更优表现。其即插即用特性可与先进强化学习策略(如NoisyRollout)无缝结合,实现互补增益。代码已开源。
原文摘要 · Abstract (English)
Despite the rapid advancements in Multimodal Large Language Models (MLLMs), a critical question regarding their visual grounding mechanism remains unanswered: do these models genuinely ``read'' text embedded in images, or do they merely rely on parametric shortcuts in the text prompt? In this work, we diagnose this issue by introducing the Visualized-Question (VQ) setting, where text queries are rendered directly onto images to structurally mandate visual engagement. Our diagnostic experiments on Qwen2.5-VL reveal a startling capability-utilization gap: despite possessing strong OCR capabilities, models suffer a performance degradation of up to 12.7% in the VQ setting, exposing a deep-seated ``modality laziness.'' To bridge this gap, we propose SimpleOCR, a plug-and-play training strategy that imposes a structural constraint on the learning process. By transforming training samples into the VQ format with randomized styles, SimpleOCR effectively invalidates text-based shortcuts, compelling the model to activate and optimize its visual text extraction pathways. Empirically, SimpleOCR yields robust gains without architectural modifications. On four representative OOD benchmarks, it surpasses the base model by 5.4% and GRPO based on original images by 2.7%, while exhibiting extreme data efficiency, achieving superior performance with 30x fewer samples (8.5K) than recent RL-based methods. Furthermore, its plug-and-play nature allows seamless integration with advanced RL strategies like NoisyRollout to yield complementary improvements. Code is available at https://github.com/aiming-lab/SimpleOCR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。