测试大模型从代码生成几何推理的能力，发现顶尖模型准确率不足50%。
GeoGramBench: Benchmarking the Geometric Program Reasoning in Modern LLMs
- 设计程序转几何任务，评估模型对代码驱动的空间推理能力。
- 17个主流大模型在最高抽象层级平均准确率低于50%。
- 适合研究符号化空间推理与几何理解的学者参考。
几何空间推理是人工智能诸多应用的基础，但大语言模型（LLMs）处理以过程代码表达的几何空间信息的能力仍缺乏深入探索。本文提出程序转几何任务，要求模型将绘制代码转化为准确且抽象的几何推理。为此，我们构建了GeoGramBench基准，包含500道经精心筛选的问题，按自定义的三级分类体系组织，侧重几何复杂度而非传统数学复杂度。对17个前沿大模型的全面评估显示，即使最先进模型在最高抽象层级准确率也低于50%。结果凸显程序驱动空间推理的独特挑战，并确立了GeoGramBench作为符号到空间几何推理研究的重要资源。项目页面：https://github.com/LiAuto-DSR/GeoGramBench。
原文摘要 · Abstract (English)
Geometric spatial reasoning forms the foundation of many applications in artificial intelligence, yet the ability of large language models (LLMs) to operate over geometric spatial information expressed in procedural code remains underexplored. In this paper, we address this gap by formalizing the Program-to-Geometry task, which challenges models to translate programmatic drawing code into accurate and abstract geometric reasoning. To evaluate this capability, we present GeoGramBench, a benchmark of 500 carefully refined problems organized by a tailored three-level taxonomy that considers geometric complexity rather than traditional mathematical reasoning complexity. Our comprehensive evaluation of 17 frontier LLMs reveals consistent and pronounced deficiencies: even the most advanced models achieve less than 50% accuracy at the highest abstraction level. These results highlight the unique challenges posed by program-driven spatial reasoning and establish GeoGramBench as a valuable resource for advancing research in symbolic-to-spatial geometric reasoning. Project page: https://github.com/LiAuto-DSR/GeoGramBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。