arXiv:2608.13766cs.CV2026-08

通过拆解图表理解能力,发现简单技能训练可大幅提升复杂推理表现。

ChartProbe: A Diagnostic Study on Visual Reasoning through Perception, Grounding, and Simple Reasoning

论文配图:ChartProbe: A Diagnostic Study on Visual Reasoning through Perception, Grounding, and Simple Reasoning
图 1 · 摘自论文原文
  • 基于图表生成代码构建诊断探针,精准定位模型弱点
  • 仅训练感知、定位和简单计算三项基础技能,复杂推理准确率显著提升
  • 无需复杂推理数据,适用于新图表类型与跨域任务

视觉语言模型在需要对图表中视觉量进行推理的问题上仍不可靠,传统观点归因于推理能力不足,并通过增加推理监督来改善。本文质疑:问题是否源于推理本身,还是源于推理所依赖的基础技能——读取图表元素(感知)、定位并绑定标签(定位)、执行排序、求和、差值等单步计算(简单推理)。为此提出ChartProbe诊断框架,其探针直接由图表渲染代码生成,保证答案精确且无需人工标注,可将失败原因精准归因到单一技能。该框架实现前人未尝试的干预:不合成复杂推理数据,而是完全剔除复杂问题与推理路径,逐项训练单一基础技能,并评估其对未见复杂推理问题的迁移效果。在三个开源权重的VLM上验证,仅通过训练这些基础技能,就能在未训练过的复杂推理任务上获得显著提升;当某项技能薄弱时,只要教会模型读图,即可在不消耗复杂推理数据的前提下恢复大部分复杂推理能力。该提升在三种分布外场景下均成立:未见过的饼图类型、独立于图像与模板的人工编写基准(ChartQA),以及非图表视觉领域(CLEVR)。因此,复杂视觉推理的性能可不依赖复杂推理监督而提升。

原文摘要 · Abstract (English)

Vision-language models (VLMs) remain unreliable on chart questions that require reasoning over visual quantities, and this weakness is usually attributed to a reasoning deficit and addressed with more reasoning supervision. We ask whether the difficulty lies in reasoning itself, or in the simpler skills that reasoning operates on: reading the plotted elements (\emph{perception}), locating them and binding them to their labels (\emph{grounding}), and performing single-step computations such as ranking, totals, and differences (\emph{simple reasoning}). We introduce \textbf{ChartProbe}, a diagnostic framework whose probes are generated directly from the code that renders each chart, so every gold answer is exact by construction, needs no human annotation, and attributes each failure to a single skill. ChartProbe enables an intervention prior work does not attempt: instead of synthesizing complex-reasoning data, we withhold complex questions and reasoning traces entirely, fine-tune on one simple skill at a time, and measure transfer to held-out complex-reasoning questions. Across three open-weight VLMs, supervising the simpler skills alone produces large gains on complex-reasoning questions the model never trained on: where these skills are weak and the model can be taught to read the image, training them recovers much of complex reasoning at no reasoning-data cost. The gains hold across three out-of-distribution settings: an unseen chart type (pie charts), a human-written benchmark disjoint from our images and templates (ChartQA), and a non-chart visual domain (CLEVR). Complex visual reasoning can therefore improve without complex-reasoning supervision.

视觉推理图表理解诊断框架基础技能

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。