测试视觉语言模型在图像中解小学数学题的能力，发现表现远低于人类。
GSM8K-V: Can Vision Language Models Solve Grade School Math Word Problems in Visual Contexts
- 将文本数学题转为多图序列，需从图像中感知并推断数量关系。
- 最强模型在新数据集上仅达59%准确率，人类达91%。
- 主要瓶颈是隐含视觉信息的推理失败，适合研究多模态推理者看。
数学推理是视觉语言模型的关键能力，但现有基准多针对文本或显式符号输入。当前尚不清楚模型能否在信息需通过图像感知和推断的场景下进行数学推理。我们提出GSM8K-V，将GSM8K转化为保留语义等价的多图像序列，通过自动化流程与人工验证，构建了1,319个高质量样本。在该数据集中，数量必须通过视觉感知提取，推理链需整合跨场景的隐含线索。对34种视觉语言模型的评估显示显著模态差距：多数模型在文本任务上超90%，但在GSM8K-V上最高仅达59%，远低于人类91%的准确率。值得注意的是，虽某些模型在已有基准上表现提升，但在GSM8K-V上未见改善，证实其衡量的是独立能力。错误分析表明，主要瓶颈在于隐含视觉推理误差（IVIE），即模型无法恢复隐含而非明示的视觉语义。代码与数据已开源。
原文摘要 · Abstract (English)
Mathematical reasoning is a key capability for vision-language models (VLMs), yet current benchmarks mainly evaluate text-based or explicitly symbolic visual inputs. It remains unclear whether VLMs can reason mathematically when information must be perceived and inferred from images rather than read from explicit symbols. We introduce GSM8K-V, a benchmark transforming GSM8K into multi-image sequences with semantic equivalence preserved. By mapping text-based problems into visual form via an automated pipeline and human verification, we curate 1,319 high-quality samples. In GSM8K-V, quantities must be extracted through visual perception, and reasoning chains must be reconstructed by integrating implicit cues across scenes. Evaluation of 34 VLMs reveals a striking modality gap: while most models exceed 90\% on text, the best model achieves only 59\% on GSM8K-V, far below the 91\% human accuracy. Notably, models enhanced for visual math reasoning show no improvement on GSM8K-V despite large gains on existing benchmarks, confirming that it evaluates a distinct capability. Error analysis shows that the primary bottleneck lies in Implicit Visual Inference Error (IVIE), where models fail to recover visual semantics that are implied rather than explicitly stated. Our code and data are released at https://github.com/ZJU-REAL/GSM8K-V.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。