构建图文逻辑推理新基准，发现大模型读图能力仍弱于纯文本。
LogicOCR: Do Your Large Multimodal Models Excel at Logical Reasoning on Text-Rich Images?
- 设计两类图文题集，覆盖生成与真实图像，评估多模态推理能力。
- 模型在图文联合推理上表现不佳，尤其面对复杂布局时准确率低。
- 提出TextCue方法，通过增强关键文本区域提升推理效果，无需重新训练。
大型多模态模型（LMMs）在视觉识别和推理方面取得进展，但其在富含文字的图像上的复杂逻辑推理能力仍不明确。为此，我们提出LogicOCR基准，包含2780道题目，分为LogicOCR-Gen（1100道选择题，基于生成图像）和LogicOCR-Real（1680道自由作答题，基于真实图像）。LogicOCR-Gen通过中国公务员考试文本构建语料库，并利用GPT-Image-1自动生成布局多样、字体丰富且视觉真实的图像，经人工验证后使用。我们在多种代表性LMM上测试了Chain-of-Thought（CoT）与直接作答两种设置。多维度分析显示：测试时缩放、输入模态差异及视觉-文本方向敏感性显著影响性能。值得注意的是，相比纯文本输入，模型在图文联合推理中仍存在明显差距。为此，我们提出TextCue——一种无需训练的方法，利用LMM注意力图与现成文本分割器定位重要文本区域，将其裁剪放大后融合至原图以增强感知。实验表明该方法有效，如在CoT设置下使LLaVA-OV-1.5-8B提升1.8%准确率。基准代码已开源。
原文摘要 · Abstract (English)
Recent advances in Large Multimodal Models (LMMs) have revolutionized their reasoning and Optical Character Recognition (OCR) capabilities. However, their complex logical reasoning performance on text-rich images remains underexplored. To bridge this gap, we introduce LogicOCR, a benchmark comprising 2780 questions with two subsets, i.e., LogicOCR-Gen with 1100 multi-choice questions on generated images, and LogicOCR-Real with 1680 meticulously designed free-form questions on real-world images. For constructing LogicOCR-Gen, we first curate a text corpus from the Chinese National Civil Servant Examination, and customize an automatic pipeline to steer GPT-Image-1 to generate images with varied layouts and fonts, ensuring contextual relevance and visual realism. Then, the generated images are manually verified. We evaluate a range of representative LMMs under Chain-of-Thought (CoT) and direct-answer settings. Our multi-dimensional analysis reveals key insights, such as the impact of test-time scaling, input modality differences, and sensitivity to visual-text orientation. Notably, LMMs still lag in multimodal reasoning compared to text-only inputs, indicating that they have not fully bridged visual reading with reasoning. Moreover, we propose TextCue, a training-free method that enhances LMMs' perception of image regions containing important text cues for solving questions. We leverage LMMs' attention maps and an off-the-shelf text segmentation specialist to determine the region, which is then cropped and enlarged to augment the original image. Experiments show its effectiveness, e.g., a 1.8% accuracy gain over LLaVA-OV-1.5-8B under the CoT setting. Our benchmark is available at https://github.com/MiliLab/LogicOCR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。