arXiv:2608.30678cs.CL2026-08

评测多模态大模型在图文理解中的元推理能力,揭示其短板与评估盲区。

OCR-MetaReasoning Benchmark: Evaluating the Meta-Reasoning Ability of MLLMs in Text-Rich Image Understanding

论文配图:OCR-MetaReasoning Benchmark: Evaluating the Meta-Reasoning Ability of MLLMs in Text-Rich Image Understanding
图 1 · 摘自论文原文
  • 将推理分为演绎、归纳、类比三类,分离答案正确性与推理过程合规性。
  • 1500个样本覆盖3种推理类型和5类文本元素,含可验证的推理路径。
  • 发现模型在规则应用和版式推理上表现弱,且正确推理过程也可能出错。

文本密集型图像理解要求多模态大语言模型(MLLMs)基于光学字符识别(OCR)结果,整合文字、布局、字段、图表及视觉对应关系中的证据。现有评估常混淆信息提取与推理,且极少检验模型是否遵循正确的推理方向:应用显式规则、抽象隐含规律或补全缺失前提。本文提出 OCR-MetaReasoning,一个受控的单图基准,将演绎、归纳、类比作为独立推理方向,并分离最终答案正确性与推理过程合规性。该基准包含1500个经验证的样本,按3×5分类法交叉覆盖三种推理类型与五类OCR对象,提供参考推理步骤、自动答案评分、元推理宏观得分(MRMS)与推理过程合规得分(RPCS)。对代表性闭源与开源MLLMs的实验表明,基于OCR的元推理尚未达到饱和:模型在显式规则应用和版式敏感推理上表现不佳,且在精确匹配评价下,过程合规的推理可能伴随错误答案。代码已公开于https://github.com/gengxuli/OCR-MetaReasoning。

原文摘要 · Abstract (English)

Text-rich image understanding requires multimodal large language models (MLLMs) to organize OCR (Optical Character Recognition)-grounded evidence across words, layout, fields, charts, and visual correspondences. Existing evaluations often conflate extraction with reasoning and rarely test whether models follow the required reasoning direction: applying visible rules, abstracting hidden regularities, or recovering missing premises. We introduce OCR-MetaReasoning, a controlled single-image benchmark that treats deduction, induction, and abduction as distinct directions and separates final-answer correctness from reasoning-process compliance. The benchmark contains 1,500 verified samples in a balanced \(3\times5\) taxonomy crossing three reasoning types with five OCR-object categories, along with reference reasoning steps, automatic answer scoring, the Meta-Reasoning Macro Score (MRMS), and the Reasoning Process Compliance Score (RPCS). Experiments with representative closed-source and open-source MLLMs show that OCR-grounded meta-reasoning remains far from saturated: models struggle with visible-rule application and layout-sensitive inference, while process-compliant rationales can accompany incorrect final answers under exact-match evaluation. The code is available at https://github.com/gengxuli/OCR-MetaReasoning.

多模态推理评估OCR

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。