用反事实图表测试视觉语言模型的真实推理能力
Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Models
- 将图表转为可执行代码，生成可控变体
- 发现模型在新图表上常无法泛化，正确率下降显著
- 适合评估模型是否真会看图推理，而非靠记忆
图表问答基准旨在提出需视觉推理才能解答的问题，但模型常通过捷径或基于背景知识对图表产生熟悉感而得出答案。为严格评估视觉推理能力，我们提出反事实图表：保持任务不变，但改变图表内容和对应答案。我们提出 Chartographer 框架，可将图表逆向工程为可执行代码，验证重建保真度，生成受控的反事实变体，并从可执行的问答逻辑中推导新答案。我们将该框架应用于现有图表问答数据集，评估专有及开源视觉语言模型（VLMs），测量其对变化的敏感性和泛化能力。反事实图表揭示了单张图表表现下隐藏的失败：模型在正确回答原图后，往往无法在新图上继续正确作答。我们发现，当更新后的图表需要全新的视觉推理路径时，失败最为普遍。
原文摘要 · Abstract (English)
Chart question-answering (QA) benchmarks aim to pose questions that require visual reasoning to correctly answer, but models can often reach solutions through shortcuts or prior familiarity with a chart based on their own background knowledge. To strictly evaluate visual reasoning, we propose counterfactual charts where the chart-question task remains fixed, but underlying chart and the corresponding answer are varied. We introduce Chartographer, a framework to reverse engineer charts into executable code, validate reconstruction fidelity, generate seed-controlled counterfactual variants, and derive new answers from executable QA logic. We apply this framework to existing chart QA datasets and evaluate proprietary and open-source vision-language models (VLMs), measuring variation sensitivity and generalizability. Counterfactual charts reveal failures hidden by single-chart performance: VLMs often fail to generalize after answering the original chart correctly. We find failures are most prevalent when updated charts require novel visual reasoning pathways.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。