视觉语言模型靠文字识别推理空间关系，纯图像则严重失准。
Can Vision-Language Models See Squares? Text-Recognition Mediates Spatial Reasoning Across Three Model Families
- 用带字和无字的方格图测试模型，发现文字能显著提升准确率
- 无文字时三模型准确率跌至60%-73%，F1降至29%-39%
- 不同模型各有失效模式，但都暴露出对非文本图像的空间定位缺陷
我们设计了一个简单实验，揭示视觉语言模型（VLMs）的根本局限：当二值网格中的填充单元缺乏文本标识时，无法准确定位。生成了十五个15×15的网格，填充密度在10.7%至41.8%之间，以两种图像形式呈现——文本符号（. 和 #）与无网格线的实心方块。让三种前沿VLM（Claude Opus、ChatGPT 5.2、Gemini 3 Thinking）进行转录。在文本符号条件下，Claude与ChatGPT的单元准确率约91%，F1达84%；Gemini准确率为84%，F1为63%。而在实心方块条件下，三者准确率均下降至60%-73%，F1降至29%-39%。关键的是，所有条件均通过相同的视觉编码器——文本符号为图像而非分词文本。文本与方块条件间的F1差距达34至54分，表明VLMs依赖高保真文字识别路径进行空间推理，其性能远超原生视觉路径。各模型在方块条件中表现出不同失效模式：系统性少计（Claude）、严重多计（ChatGPT）、模板幻觉（Gemini），但共有的核心缺陷是：对非文本视觉元素的空间定位能力严重退化。
原文摘要 · Abstract (English)
We present a simple experiment that exposes a fundamental limitation in vision-language models (VLMs): the inability to accurately localize filled cells in binary grids when those cells lack textual identity. We generate fifteen 15x15 grids with varying density (10.7%-41.8% filled cells) and render each as two image types -- text symbols (. and #) and filled squares without gridlines -- then ask three frontier VLMs (Claude Opus, ChatGPT 5.2, and Gemini 3 Thinking) to transcribe them. In the text-symbol condition, Claude and ChatGPT achieve approximately 91% cell accuracy and 84% F1, while Gemini achieves 84% accuracy and 63% F1. In the filled-squares condition, all three models collapse to 60-73% accuracy and 29-39% F1. Critically, all conditions pass through the same visual encoder -- the text symbols are images, not tokenized text. The text-vs-squares F1 gap ranges from 34 to 54 points across models, demonstrating that VLMs behave as if they possess a high-fidelity text-recognition pathway for spatial reasoning that dramatically outperforms their native visual pathway. Each model exhibits a distinct failure mode in the squares condition -- systematic under-counting (Claude), massive over-counting (ChatGPT), and template hallucination (Gemini) -- but all share the same underlying deficit: severely degraded spatial localization for non-textual visual elements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。