用平面图绘图测试大模型空间推理能力，发现约束数比顶点数更决定难度。
PlanarBench: Evaluating LLM Spatial Reasoning via Planar Graph Drawing
- 让模型仅凭边列表画无交叉的平面图，考察空间布局能力
- 边数与得分相关性达-0.85，远高于顶点数的-0.47
- 适合评估模型几何推理能力，尤其关注复杂布局生成
现有大模型图推理基准多要求回答图论问题或符号计算，而非构建空间布局。任务难度通常按顶点数划分，但研究表明难度更受边约束数量影响。本文提出PlanarBench，要求模型根据边列表生成无交叉的平面图ASCII图示。在91种模型配置和199个2-7个顶点的连通非同构平面图上，边数与平均得分的相关性（r = -0.85）显著高于顶点数（r = -0.47），且在控制顶点数后仍保持强相关性（r_p = -0.80）。该基准实现了对两种难度维度的分离。此外，绘图面积和输出长度与得分无显著相关性，排除了单纯输出规模的影响。性能差异显著：最优模型得分为159.5/199，多数小于30B参数的模型得分低于25，前沿系统仍存在大量失败案例。
原文摘要 · Abstract (English)
Existing LLM graph benchmarks typically ask models to answer graph-theoretic questions or compute symbolic solutions rather than construct spatial layouts. Within-task difficulty is also primarily stratified by vertex count. However, existing research also suggests that task difficulty is more closely related to the number of constraints imposed by the edges than to the number of vertices being arranged. We introduce PlanarBench, a benchmark that asks models to produce crossing-free ASCII drawings of planar graphs given only an edge list. Across 91 model configurations and 199 non-isomorphic connected planar graphs with 2-7 vertices, edge count is more strongly associated with mean task score than vertex count ($r=-0.85$) versus ($r=-0.47$) and remains strongly associated after controlling for vertex count ($r_p=-0.80$). PlanarBench provides a controlled setting for separating these two difficulty axes. In addition, neither drawing area nor total response length demonstrated a meaningful correlation with score, which is evidence against a simple output-size explanation. Performance varies widely: the best model scores 159.5 out of 199, most models below 30B parameters score under 25, and substantial failures remain among frontier systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。