构建视觉编程评测基准，揭示大模型在几何代码生成上的短板
TurtleAI: Benchmarking Multimodal Models for Visual Programming in Turtle Graphics
- 设计823个真实图形编程任务，测试模型对几何模式的感知与代码生成能力
- 主流模型成功率普遍低于30%，尤其在空间推理和精准复现上表现差
- 通过少量种子样本生成合成数据，微调后提升约20%性能，适合教育场景研究
视觉语言模型（VLMs）已被用于视觉编程，即根据图像生成解决视觉任务的代码。然而，以往研究多聚焦于提高生产力，尚不清楚当前VLMs在面向教育的视觉编程任务中的表现如何，以及制约其性能的关键因素。为填补这一空白，我们提出TurtleAI，一个包含823个任务的基准，这些任务基于真实世界中的Turtle Graphics编程场景精心设计。完成这些任务需模型理解几何模式、推理空间关系，并生成准确还原几何图形的Python代码。我们评估了20多个VLM，包括GPT-5、GPT-4o和Qwen2-VL-72B，发现它们表现不佳，大多数成功率达不到30%。为改善此问题，我们提出一种仅需少量种子样本的数据生成方法。将Qwen2-VL-72B在生成的合成数据上微调后，在真实任务上性能提升约20%。失败分析表明，GPT-4o在空间推理和精确视觉复现方面存在缺陷，而微调主要提升了视觉推理与代码实现之间的对齐度。
原文摘要 · Abstract (English)
Vision-language models (VLMs) have been explored for visual programming, where they generate code to solve visual tasks. However, most prior work focuses on visual programming for productivity; it remains unclear how well current VLMs perform on education-oriented visual programming and what factors limit their performance. To bridge this gap, we introduce TurtleAI, a benchmark containing 823 tasks curated based on real-world visual programming tasks in the Turtle Graphics domain. Solving these tasks requires models to perceive geometric patterns, reason about spatial relationships, and synthesize Python code that faithfully reproduces geometric patterns. We evaluate 20+ VLMs, including GPT-5, GPT-4o, and Qwen2-VL-72B, and find that they struggle significantly, with most achieving success rates below 30%. To address these limitations, we propose a data generation technique that requires only a small set of seed samples. Fine-tuning Qwen2-VL-72B on the resulting synthetic data yields an improvement of about 20% on real-world tasks. Our failure analysis reveals that GPT-4o struggles with spatial reasoning and precise visual replication, whereas fine-tuning primarily improves the alignment between visual reasoning and code implementation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。