arXiv:2608.30751cs.AIcs.CV2026-08

测试文本模型画图能力,发现布局表现受模型和输出方式影响。

Autoregressive Mosaics: Probing 2D Spatial Reasoning in Text-Only Language Models

论文配图:Autoregressive Mosaics: Probing 2D Spatial Reasoning in Text-Only Language Models
图 1 · 摘自论文原文
  • 设计双任务基准,区分图像描述翻译与自由布局生成
  • 所有模型都能准确转译指定几何,但自由布局能力差异大
  • 用SVG代替代码作输出,布局效果普遍提升

仅用文本和代码训练的大语言模型有时能生成可识别图像的程序。然而,这反映的是对二维空间布局的内部表征,还是仅能将空间描述转为代码尚不明确。我们提出自回归马赛克(AM-Bench)基准,包含两类任务:一是将完整几何描述转为代码;二是根据不完整提示生成图像。在八种开源文本-代码模型上,所有模型均能可靠完成翻译任务,但自由布局表现差异显著,说明布局能力不能仅由代码生成能力解释。进一步消融实验表明,输出媒介至关重要:将程序代码替换为原始SVG后,所有模型的布局得分均提升。探针分析显示,生成前存在粗略布局计划,但仅反映提示中的隐含布局;生成过程中,模型追踪动态几何状态而非执行固定计划。总体表明,文本模型的二维空间表现取决于模型自身与输出媒介,且不单由代码生成能力决定。

原文摘要 · Abstract (English)

Large language models (LLMs) trained only on text and code can sometimes generate programs that draw recognizable images. However, it is unclear whether this reflects an internal representation of 2D spatial layout or simply the ability to translate spatial descriptions into code. We introduce Autoregressive Mosaics (AM-Bench), a benchmark that separates these factors: First, a translation task gives a model a fully specified geometry of a picture in words as a prompt and asks for the code that produces it. Second, a layout task requires the model to compose an image from an underspecified prompt. Across eight open-weight text-and-code-only models, all models reliably translate specified geometry into code, but their open-ended layout performance differs substantially, indicating that these differences are not explained by code-generation ability alone. An output-medium ablation further shows that the interface or medium of expression that the model uses matters: replacing procedural code with raw SVG improves layout scores across all models. Finally, probing model activations shows that a coarse layout plan is present before generation, but reflects only the layout implied by the prompt. During generation, models track the evolving geometric state instead of executing an initially fixed plan. Overall, these results show that 2D spatial performance in text-only LLMs depends on both the model and the output medium, and is not explained by code-generation ability alone.

空间推理模型评估生成式模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。