MLLM看图推理差,根源是看不懂图,靠文字凑答案。
Math Blind: Failures in Diagram Understanding Undermine Reasoning in MLLMs
- 用诊断测试分离看图和推理,发现模型看图能力极弱。
- 细粒度定位准确率接近零,图形理解严重依赖文字线索。
- 将图转为结构化关系图,可让推理提升3-4%,适合数学多模态研究者。
图表是一种通过结构化符号及其空间排列编码抽象概念与关系的视觉语言,具有高度人工性与符号性,区别于自然图像,对多模态大模型(MLLMs)构成独特挑战。近期研究发现,MLLM在处理图表时常出现错误推理与幻觉。本文探究其根源是否在于模型本身对图表的感知能力不足。为此,我们构建了一套诊断测试集,将感知与推理任务分离。系统评估显示,MLLM在基础感知任务上表现不佳,如形状分类、物体计数、关系识别和物体定位,细粒度定位准确率接近零。进一步分析表明,模型因视觉感知薄弱而产生‘文本盲信’,即过度依赖文本线索而非真实视觉理解(即‘数学视盲’)。我们假设,若让模型学习图表的内在结构特性——以基元及其相互关系构成的图表示——将有助于提升理解能力。7B与32B规模的实验验证该假设:基于此类表示训练的模型,在定位任务上提升79%。更重要的是,这种改进可迁移至推理任务,在三个公开基准上实现3-4%的跨套件性能提升,且无需额外链式推理数据。结果表明,低层感知支持高层推理的忠实性。本文提供了方法框架与实证证据,指导未来相关研究。
原文摘要 · Abstract (English)
Diagrams represent a form of visual language that encodes abstract concepts and relationships through structured symbols and their spatial arrangements. Unlike natural images, they are inherently symbolic, and entirely artificial. They thus pose unique challenges for Multimodal Large Language Models (MLLMs) distinct from natural image processing. Recent studies have shown that MLLMs often exhibit flawed reasoning and hallucinations when handling diagram inputs. We investigate here whether these limitations stem from shortcomings in the models' ability to interpret diagrams themselves. To this end, we develop a diagnostic test suite that isolates perception from reasoning. Our systematic evaluation reveals that MLLMs perform poorly on basic perceptual tasks, e.g., shape classification, object counting, relationship identification, and object grounding, with near-zero accuracy on fine-grained grounding. Further analysis shows that weak diagram perception leads to "blind faith in text", where models rely on textual shortcuts rather than visual understanding (that is, they are Math Blind). We hypothesize that enabling models to capture the inherent structural properties of diagrams, represented as graphs of primitives and their interrelationships, is essential for improving diagram understanding. Experiments with 7B and 32B MLLMs validate this assumption, with models trained on such representations achieving a +79% gain on the grounding task. Crucially, these gains transfer to reasoning, achieving 3-4% cross-suite improvements on three public benchmarks even without additional chain-of-thought reasoning data. Our findings demonstrate that low-level perception supports faithful high-level reasoning in mathematical MLLMs. We provide both methodological frameworks and empirical evidence to guide future research in this direction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。