评测大模型画几何图的能力,发现解题强不等于画图准。
Solving Is Not Drawing: A Benchmark for Diagrammatic Reasoning in Olympiad Geometry

- 构建954道奥数几何题的图文数据集,含高保真Asymptote绘图代码
- 大模型绘图编译成功率仅36.14%,远低于解题正确率
- 首次系统评测模型的图形推理能力,适合研究具身认知与视觉推理者
GPT、Claude等基础模型在奥数数学题上表现出色,几何解题已成为衡量其数学推理能力的标准。然而,解题与绘制准确几何图并非同一能力:正确解题往往依赖于包含恰当辅助线和交点关系的高保真图示,而现有模型虽能推理出答案,却未必能生成有效图形。尽管已有MathVista、MathVerse等基准测试解题正确性,但尚无评估绘图能力的独立指标。本文提出一个开源基准,包含954个自包含的奥数几何题(其中297题为难题),每题配有人工撰写的高保真图示(可运行的Asymptote代码)及解题过程,并设计了文本、代码、图像、视觉语言模型及约束匹配等多维度度量方法,以评估所谓的“图形推理”能力。对当前大模型的评估显示,其绘图准确性显著不足,平均编译成功率为36.14%。结果表明,强大的数学推理能力并不等同于准确的图形构造能力。数据集与基准可通过https://huggingface.co/datasets/max98765/hard_geometry_problems_with_diagrams获取。
原文摘要 · Abstract (English)
Foundation models such as GPT and Claude now solve olympiad-level mathematics with remarkable proficiency, so much so that geometry problem solving has become a standard proxy for their mathematical reasoning. Yet solving a geometry problem and drawing the figure it depends on are not the same skill: progress often hinges on a faithful diagram with the right auxiliary constructions and incidences, and it is unclear that a model which reasons its way to the answer can also produce one. A growing collection of benchmarks, including MathVista, and MathVerse, measures whether models reach the correct answer, but to our knowledge, none isolate the distinct ability to construct the diagram itself, leaving this capability unmeasured. We introduce an open-source benchmark that targets this gap: 954 self-contained olympiad geometry problems, with a 297-problem hard subset, each paired with its solution and a human-authored, high-fidelity diagram in renderable Asymptote code, together with a suite of text-, code-, image-, VLM-, and constraint-based metrics for what we term diagrammatic reasoning. Evaluating current foundation models reveals a pronounced gap between solving and drawing: their diagrams are markedly less faithful, with an average compile success rate of only 36.14\%. Strong mathematical reasoning, we find, does not imply the ability to construct accurate geometric diagrams. Our benchmark and dataset can be accessed at https://huggingface.co/datasets/max98765/hard_geometry_problems_with_diagrams.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。