评测大模型在图形编程中的几何理解与代码生成能力
TurtleBench: A Visual Programming Benchmark in Turtle Geometry
- 基于龟形几何设计视觉编程任务，测试模型从图像或文字生成代码的能力
- GPT-4o在最简单任务上准确率仅19%，少样本提示提升不足2%
- 适合研究视觉-代码联合推理、具身认知与生成式模型评估的学者
人类自幼便具备识别图像与场景中几何模式的能力，但当前大型多模态模型（LMMs）在类似推理方面仍面临挑战，亟需可靠的评估方法。本文提出 \\(Turtle\\) 基准，用于评估 LMMs 在给定视觉示例、文本指令或二者结合时，解读几何模式并生成精确代码的能力。该基准受龟形几何启发，设计了一系列具有算法逻辑的图案任务。评估显示，主流 LMMs 表现显著不足：GPT-4o 在最简单任务上准确率仅为 19\\%；少样本提示带来的性能提升不足 2\\"。\\Turtle 揭示了人类与人工智能在直观视觉几何理解上的巨大差距，是少数专门评估视觉理解与代码生成融合能力的基准之一，为未来研究奠定基础。代码与数据集详见：\\href{https://github.com/sinaris76/TurtleBench}{https://github.com/sinaris76/TurtleBench}
原文摘要 · Abstract (English)
Humans have the ability to reason about geometric patterns in images and scenes from a young age. However, developing large multimodal models (LMMs) capable of similar reasoning remains a challenge, highlighting the need for robust evaluation methods to assess these capabilities. We introduce \Turtle, a benchmark designed to evaluate LMMs' capacity to interpret geometric patterns -- given visual examples, textual instructions, or both -- and generate precise code outputs. Inspired by turtle geometry, a notion used to teach children foundational coding and geometric concepts, TurtleBench features tasks with patterned shapes that have underlying algorithmic logic. Our evaluation reveals that leading LMMs struggle significantly with these tasks, with GPT-4o achieving only 19\% accuracy on the simplest tasks and few-shot prompting only marginally improves their performance ($<2\%$). \Turtle highlights the gap between human and AI performance in intuitive and visual geometrical understanding, setting the stage for future research in this area. \Turtle stands as one of the few benchmarks to evaluate the integration of visual understanding and code generation capabilities in LMMs, setting the stage for future research. Code and Dataset for this paper is provided here: \href{https://github.com/sinaris76/TurtleBench}{https://github.com/sinaris76/TurtleBench}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。