构建可更新的合成形状场景感知到代码重建基准，测试模型从图像还原可执行绘图程序的能力。
ShapeCodeBench: A Renewable Benchmark for Perception-to-Program Reconstruction of Synthetic Shape Scenes
- 基于随机种子生成新样本，避免数据泄露，支持持续评估
- 当前最强模型仅在简单场景中达90%以上精确匹配，复杂重叠场景性能骤降
- 适合研究多模态模型、程序逆向与视觉-语言对齐的学者使用
我们提出ShapeCodeBench，一个用于感知到程序重建的合成基准：给定一张渲染的位图图像，模型需输出一个可执行的绘图程序，由确定性评估器重新渲染并对比目标。v1版领域特定语言（DSL）在512×512黑白画布上包含四个基本图形原语，每个实例由带种子的随机数生成器生成，因此可不断创建新的保留集以减少实例污染。我们发布了一个冻结的eval_v1划分，包含150个样本，分为易、中、难三类，评分指标包括精确匹配、像素准确率、前景交并比、解析成功率和执行成功率。评估了空程序基线、经典计算机视觉启发式方法、Claude Opus 4.7（高/最大努力）、GPT-5.5（中/超高级推理）。启发式方法在简单场景表现良好，但重叠融合时性能崩溃；最强多模态配置仍能保持大部分前景结构，但因微小参数误差无法达成精确匹配。总体精确匹配率仍很低，表明该基准尚未饱和。相关代码、冻结数据集、运行结果及论文源码均已公开，支持独立复现与扩展。
原文摘要 · Abstract (English)
We introduce ShapeCodeBench, a synthetic benchmark for perception-to-program reconstruction: given a rendered raster image, a model must emit an executable drawing program that a deterministic evaluator re-renders and compares with the target. The v1 DSL has four primitives on a 512 x 512 black-on-white canvas, but every instance is generated from a seeded RNG, so fresh held-out sets can be created to reduce exact-instance contamination. We release a frozen eval_v1 split with 150 samples across easy, medium, and hard tiers, scored by exact match, pixel accuracy, foreground IoU, parse success, and execution success. We evaluate an empty-program floor, a classical computer-vision heuristic, Claude Opus 4.7 at high and max effort, and GPT-5.5 at medium and extra_high reasoning effort. The heuristic is competitive on easy scenes but collapses when overlaps fuse components; the strongest multimodal configuration preserves much of the foreground structure but still misses exact match because of small parameter errors. Best overall exact match remains low, so ShapeCodeBench is far from saturated. The benchmark code, frozen dataset, run artifacts, and paper sources are released to support independent replication and extension.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。